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PREFACE 


This  study  presents  the  combined  efforts  of  three  military  Research  Fellows,  participating  in  an 
11-month  Defense  Systems  Management  College  Research  Fellowship  program,  sponsored  by 
the  Under  Secretary  of  Defense  for  Acquisition  and  Technology.  In  keeping  with  its  role  as  the 
center  for  systems  management  education  in  the  Department  of  Defense  (DoD),  the  Defense 
Systems  Management  College  (DSMC)  conducts  this  annual  fellowship  program  to  research  a 
subject  of  vital  interest  to  the  U.S.  defense  acquisition  community. 

To  achieve  program  objectives,  program  managers  have  long  been  applying  modeling  and  simu¬ 
lation  (M&S)  tools  to  efforts  within  the  various  stages  of  their  programs.  Recently,  however, 
declining  defense  budgets  have  increased  the  pressure  on  the  acquisition  community  to  find 
cheaper  ways  to  develop  and  field  systems.  Additionally,  the  rapid  pace  of  changing  world  events 
demands  that  these  material  solutions  get  into  the  hands  of  the  warfighter  faster.  To  meet  the 
increased  challenge  of  budget  and  time  constraints,  many  programs  have  radically  changed  the 
way  they  conduct  business.  These  programs  recognize  the  powerful  increases  in  productivity 
and  decreases  in  cost  brought  by  M&S  tools.  Within  these  programs,  program  management 
looks  to  weave  M&S  applications  across  program  phases  and  seeks  to  leverage  the  strengths  of 
external  M&S  applications  to  efforts  within  their  program.  This  new  way  of  doing  business, 
coupling  rapid  advances  in  simulation  technology  with  process  change,  is  fueling  a  new  approach 
to  how  we  acquire  defense  systems.  This  new  approach  is  being  termed  Simulation  Based 
Acquisition,  or  SBA. 

Objective  of  Study 

The  objective  of  this  book  is  to  convince  program  managers  that  SBA  is  a  smarter  way  of  doing 
business.  We  will  do  this  by  defining  SBA,  explaining  the  strengths  of  SBA,  and  describing  the 
forces  that  will  encourage  its  use.  Where  possible,  we  highlight  best  practices  and  useful 
implementation  guidance. 

Within  the  DoD,  there  are  a  staggering  number  of  variables  that  an  acquisition  program  office 
must  evaluate  and  analyze.  There  is  an  almost  infinite  number  of  possible  applications  of  SBA 
activities  within  DoD  acquisition  programs.  Where  they  apply,  we  present  examples  of  com¬ 
mercial  applications  of  SBA.  Most,  however,  are  narrowly  focused  and  are  of  questionable 
general  use  to  acquisition  program  offices.  This  is  partly  because  acquisition  programs  within 
the  DoD  are  unique  in  their  complexity  compared  with  many  commercial  enterprises.  Systems 
that  the  DoD  produces  are  usually  composed  of  many  varied  sub-components  that  push  the 
boundaries  of  technology.  When  these  complex  sub-components  are  brought  together  in  the 
aggregate  at  the  system  level,  the  complexity  of  the  program  is  compounded.  We  hope  that  this 
“round  down  range”  will  stimulate  discussion  and  provide  the  mark  from  which  to  “adjust  fire.” 
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Methodology 


We  conducted  our  research  in  four  areas.  First,  we  embarked  on  basic  research  and  information 
collection  while  attending  the  12-week  residential  Program  for  Management  Development 
(PMD)  course  at  the  Harvard  University  Graduate  School  of  Business.  Second,  we  conducted 
an  extensive  search  of  the  applicable  literature.  Third,  we  conducted  interviews  and  attended 
briefings  and  conferences  on  Simulation  Based  Acquisition.  Finally,  we  held  two  in-process 
reviews  with  students  attending  the  DSMC  Advanced  Program  Managersment  Course  (APMC), 
and  with  the  DSMC  faculty. 

Our  initial  research  began  while  we  were  at  the  Harvard  Business  School.  There,  we  presented 
and  discussed  issues  concerning  SBA  with  faculty  members  and  our  fellow  classmates.  As  our 
classmates  (157  students  from  38  countries)  represented  both  U.S.  and  international  compa¬ 
nies,  we  gained  truly  global  insights  into  a  few  of  the  topic  areas.  We  were  also  fortunate  to  have 
a  few  classmates  working  for  U.S.  defense  contractors,  who  provided  valuable  perspectives  into 
government  and  industry  interrelationships  and  model  sharing.  Our  discussions  centered  on 
applicable  business  practices  and  answers  to  numerous  questions  raised  by  our  research  topic. 
For  example,  should  program  managers  be  provided  incentives  to  design  and  develop  models 
and  simulations  that  allow  for  reuse  and/or  integration  into  other  programs?  Are  there  appli¬ 
cable  business  practices,  or  measures  of  success/metrics  that  could  evaluate  the  effective  use  of 
a  government  program’s  modeling  and  simulation  efforts?  Where  did  they  see  technology 
going  in  the  near  future? 

Our  second  area  of  research  was  a  comprehensive  literature  review  and  Internet  search  cover¬ 
ing  the  topical  areas  of  modeling  and  simulation  and  Simulation  Based  Acquisition.  A  particu¬ 
larly  useful  area  was  the  SBA  Special  Interest  Group  on  the  Defense  Modeling  and  Simulation 
Office’s  World  Wide  Web  home  page  (www.dmso.mil),  which  contains  a  lot  of  historical  and 
current  information  on  the  subject,  as  well  as  up-to-date  links  to  modeling  and  simulation 
organizations  and  groups  which  are  active  on  the  web. 

Our  third  area  of  research,  conducting  interviews  and  attending  briefings  and  conferences, 
provided  most  of  our  information.  We  broke  our  areas  of  emphasis  into  three  categories:  gov¬ 
ernment,  defense  industry  and  commercial  industry.  To  gain  insight  into  government  program¬ 
matic  issues,  we  visited  Service  Acquisition  Offices,  acquisition  and  test  organizations,  newly 
formed  program  offices,  and  established  program  offices.  We  obtained  an  understanding  of  the 
defense  industry’s  support  to  government  programs  through  visits  to  corporation  headquarters 
and  contractor  facilities.  We  visited  commercial  firms  that  have  been  making  significant 
investments  in  simulation  technology.  In  all,  we  conducted  over  85  interviews  (see  Appen¬ 
dix  B).  Some  interviews  were  as  short  as  thirty  minutes,  while  some  lasted  over  the  course 
of  three  days.  Most,  however,  were  three  or  four  hours  in  duration.  The  level  of  the  interviewees 
varied  greatly,  from  Senior  Acquisition  Officials  and  Program  Managers  to  individuals  tasked 
with  constructing  physical  wooden  mock-ups  (used  in  the  verification  of  virtual  models).  Though 
the  sources  varied,  there  was  a  great  deal  of  commonality  in  the  views  expressed.  We  also 
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participated  in  and  attended  several  SBA  conferences  and  workshops.  Each  site  visited  provided 
unique  insights  into  the  collage  that  is  the  Simulation  Based  Acquisition  picture. 

Our  final  area  of  data  collection  was  through  peer  and  faculty  review  at  DSMC.  Through  frank 
discussions  conducted  in  our  office  spaces  and  the  use  of  the  DSMC  s  Management  Delibera¬ 
tion  Center  (a  Group  Decision  Support  System),  many  of  the  APMC  students  provided  us  with 
an  excellent  sounding  board  on  the  direction  and  progress  of  our  research.  These  in-process 
reviews  helped  ensure  we  were  addressing  the  issues  most  important  to  the  acquisition  commu¬ 
nity  concerning  Simulation  Based  Acquisition. 

Special  Thanks 

The  Research  Fellows  extend  a  special  note  of  appreciation  to  Ms.  Joan  Sable,  DSMC  Military 
Research  Fellowship  Coordinator.  Ms.  Sable  ensured  that  our  administrative  and  logistical 
requirements  were  met  at  DSMC  and  Harvard,  and  her  support  enabled  us  to  concentrate  our 
attention  and  energies  on  the  research  and  writing  of  this  report. 

For  all  of  their  guidance  throughout  our  research  project  we  pay  special  thanks  to  Colonel 
Kenneth  “Crash’'  Konwin,  Director  Defense  Modeling  and  Simulation  Office;  Ms.  Robin  Frost, 
Office  of  the  Secretary  of  Defense  (Director  of  Test,  Systems  Engineering,  and  Evaluation); 
and  Mr.  Steve  Olson  and  the  other  members  of  the  National  Defense  Industrial  Association’s 
SBA  Industry  Steering  Group. 

We  appreciate  the  efforts  of  the  DSMC  Press  staff  for  their  many  hours  working  on  this  report 
to  ensure  its  highest  quality.  Thanks  to  the  Visual  Arts  and  Press  staff  for  their  work  on  the 
graphs,  charts  and  cover  page  as  well  as  their  many  hours  in  the  layout  of  this  report.  Finally,  we 
extend  a  special  thank  you  to  Air  Force  Academy  Cadets  First  Class  Paul  Ferguson  and  Nathan 
Atherley  for  their  research  assistance  during  their  summer  internship  at  DSMC. 

There  are  others,  too  numerous  to  mention  individually,  who  deserve  recognition.  The  three 
Research  Fellows  would  like  to  thank  all  of  those  interviewed.  As  a  token  of  our  appreciation, 
we  dedicate  this  effort  to  you.  May  our  report  be  as  helpful  to  you  as  you  were  to  us. 
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INTRODUCTION 


The  Services  and  DoD  are  committed  to  pro¬ 
vide  superior  weapon  systems  and  materials 
to  the  warfighter  faster,  and  at  less  cost.  To 
achieve  this,  the  acquisition  community  is 
charged  with  providing  “better,  faster,  and 
cheaper”  material  solutions.  Our  research 
shows  that  the  practices  and  processes  of  simu¬ 
lation  based  acquisition  (SBA)  will  help  over¬ 
come  many  of  the  hurdles  associated  with 
acquiring  “better,  faster,  and  cheaper” 
solutions. 

The  phrase  “better,  faster,  and  cheaper”  is 
deceptively  simple,  because  the  execution  of 
this  task  across  the  Services  and  DoD  is  a  com¬ 
plex  undertaking.  As  many  program  offices 
will  attest,  this  statement  rings  true  for  three 
reasons.  First,  the  current  Government  acqui¬ 
sition  process,  with  its  oversight  requirements 
and  the  nature  of  its  funding,  is  not  the  most 
streamlined  or  efficient  of  processes.  Second, 
many  large  Government  acquisition  programs 
develop  complex  systems,  which  push  the  lim¬ 
its  of  technology.  And  third,  program  complex¬ 
ity  is  further  magnified  since  most  of  the  ma¬ 
terial  solutions  produced  are  not  stand-alone 
products.  That  is,  they  are  “force  multipliers” 
that  must  interface  with  and  enhance  the  com¬ 
bat  performance  of  other  systems  that  are  ei¬ 
ther  fielded  or  in  development.  Unlike  many 
commercial  programs,  therefore,  the  current 


acquisition  processes  (indeed  the  very  solutions 
themselves)  are  usually  complicated. 

To  be  successful,  a  program  office  must  bal¬ 
ance  the  complex  relationships  that  exist 
between  “better”,  “faster”,  and  “cheaper.” 
The  degree  of  balance  is  not  usually  measured 
directly,  but  it  can  be  measured  in  terms  of 
the  risk  in  meeting  objectives.  Risk  is  a  mea¬ 
sure  of  the  inability  to  achieve  a  program’s 
defined  performance,  schedule,  and  cost 
objectives.  It  has  two  components:  the  prob¬ 
ability  of  failing  to  achieve  particular  perfor¬ 
mance,  schedule,  or  cost  objectives  and  the 
consequences  of  failing  to  achieve  those 
objectives.^  SBA  can  address  the  first  compo¬ 
nent  of  risk  by  increasing  the  likelihood  of  pro¬ 
ducing  systems  that  have  “better”  performance, 
“faster”  schedule,  and  “cheaper”  cost. 

Better  Performance 

No  one  would  advocate  providing  the 
warfighter  with  something  that  is  “almost  as 
good  as.”  The  material  solutions  that  the  U.S. 
defense  acquisition  community  produces  are 
the  tools  of  victory  on  future  battlefields.  Find¬ 
ing  and  producing  “better”  solutions  starts 
with  the  warfighter’s  ability  to  correctly  articu¬ 
late  the  requirement.  This  frames  all  of  the 
acquisition  activities  that  follow. 
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Concepts  that  satisfy  stated  requirements  must 
then  be  identified  and  evaluated.  The  military 
has  a  much  harder  problem  to  solve  than  does 
the  commercial  automotive  or  aviation  indus¬ 
tries,  since  they  usually  have  a  good  starting 
point.  And  while  they  produce  some  very  com¬ 
plicated  products,  the  basic  concepts  for  com¬ 
mercial  products  do  not  deviate  greatly  from 
one  generation  to  the  next.  Follow-on  systems 
within  the  DoD,  on  the  other  hand,  are  usu¬ 
ally  radical  departures  from  their  predeces¬ 
sors.  Take  as  an  example,  air  superiority  fight¬ 
ers;  the  F-22  can  hardly  be  viewed  as  a  variant 
of  the  F-15.  The  increased  performance 
demanded  of  follow-on  DoD  systems  brings 
with  it  the  increased  likelihood  that  these 
performance  objectives  will  not  be  met. 

Through  the  use  of  modeling  and  simulation, 
an  SBA  process  offsets  this  increased  perfor¬ 
mance  risk  in  three  ways.  First,  it  enables  the 
warfighter  to  become  a  member  of  the  design 
team  and  to  influence  the  design  much  earlier 
than  the  current  process  allows.  Simulation 
provides  the  tools  for  the  warfighter  to  visual¬ 
ize  and  interact  with  the  system  to  perform 
operational  analyses  and  assessments.  The 
design  team  rapidly  incorporates  necessary 
changes  into  the  design  based  upon  this  expert 
input  and  the  result  is  a  better  solution.  Sec¬ 
ond,  the  SBA  process  provides  rapid  feedback 
to  the  design  team  by  enabling  them  to  per¬ 
form  “what  if”  analyses,  or  iterations,  on 
hundreds  of  point  designs.  Rather  than  build¬ 
ing  a  physical  prototype  to  validate  a  single 
point  design,  designers  can  use  a  virtual  pro¬ 
totype  to  look  at  potentially  hundreds  of 
design  variations.  The  rapid  feedback  and 
learning  from  these  iterations  will  enable  the 
program  office  to  produce  a  better  system. 
Third,  not  only  are  we  able  to  conduct  more 
iterations,  but  we  are  also  able  to  converge  on 
more  optimal  designs  because  we  can  test 


many  of  our  assumptions  to  see  if  they  are  true. 
Furthermore,  as  the  tools  for  looking  at  the 
entire  life  cycle  of  the  system  improve,  design 
teams  will  be  able  to  move  from  form  and  fit 
issues  to  the  more  complex  issues  of  function. 
This,  too,  will  contribute  to  better  solutions. 

Faster  Schedule 

The  military  goes  to  war  with  what  is  fielded, 
not  with  what  is  on  the  drawing  boards  or  in 
the  acquisition  pipeline.  As  Lieutenant  Gen¬ 
eral  Paul  J,  Kern,  Director  of  the  Army's 
Acquisition  Corps  stated,  “The  current  acqui¬ 
sition  process  was  good  for  producing  systems 
in  the  Cold  War  environment,  where  we  had  a 
predictable  enemy  with  known  lead  times. 
Now,  many  of  our  foreseeable  potential 
enemies  are  different;  they  are  not  constrained 
by  a  rigid,  inflexible  acquisition  process.  They 
can  purchase  weapon  systems  and/or  sub-com¬ 
ponents  in  an  open-air  market  environment, 
like  a  global  off-the-shelf  system.  Through 
mixing  and  matching  various  weapon  systems 
and  subsystems,  they  can  rapidly  generate  some 
very  lethal  systems.  We  lose  if  they  can  purchase 
and  bring  together  their  systems  faster  than 
we  can  develop  ours  because  of  long  cycle 
times.  ”2 

By  enabling  the  acquisition  process  to  get 
inside  the  adversary’s  decision  cycle,  an  SBA 
process  increases  the  likelihood  of  develop¬ 
ing  and  producing  systems  “faster”.  If  we  can 
model  what  we  need  answers  to,  we  can  get 
rapid  feedback  through  the  power  of  simula¬ 
tion.  The  program  gets  the  required  informa¬ 
tion  much  sooner  in  the  process  than  it  does 
under  the  current  system,  translating  into 
faster  cycle  times.  Additionally,  we  can  con¬ 
duct  many  more  processes  concurrently 
because  more  people  can  access  the  informa¬ 
tion  they  need  at  the  same  time.  Concurrent 
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engineering  activities  within  an  SBA  process 
will  compress  the  cycle  time  of  weapons  acqui¬ 
sition.  Working  with  virtual  prototypes  and  digi¬ 
tal  product  descriptions  allows  program  man¬ 
agers  to  exhibit  agility  in  designing  and  evalu¬ 
ating  concepts.  For  example,  using  computa¬ 
tional  fluid  dynamics,  the  test  community  can 
help  the  design  team  look  at  weapons  release 
issues  much  earlier  and  at  a  fraction  of  the  cost 
of  running  a  wind  tunnel  test.  Virtual  manu¬ 
facturing  allows  the  manufacturing  community 
to  look  at  procedures  such  as  reducing  the 
number  of  parts  in  the  design  as  well  as  re¬ 
hearsing  production  processes.  Using  three- 
dimensional  solid  modeling,  shop-floor  me¬ 
chanics  can  assess  the  maintainability  of  the 
design  and  make  design  inputs  alongside  the 
engineers,  much  earlier  in  the  process. 

Cheaper  Cost 

“Cheaper”  can  be  broken  down  into  two  areas. 
The  first  area  involves  reducing  the  initial 
acquisition  cost  of  systems.  This  point  was 
illustrated  by  Norm  Augustine  two  decades 
ago  in  one  of  his  famous  Augustine’s  Laws: 
“In  the  year  2054,  the  entire  defense  budget 
will  purchase  just  one  tactical  aircraft.  This  air¬ 
craft  will  have  to  be  shared  between  the  Air 
Force  and  Navy  3  days  each  per  week.”^ 
Clearly  there’s  a  need  to  reverse  and  stabilize 
the  trend  in  rising  initial  acquisition  costs.  Sec¬ 
ond,  “cheaper”  also  includes  reducing  the  an¬ 
nual  costs  of  operating  and  sustaining  systems 
until  and  including  disposal.  The  foreseeable 
trend  in  system  life  span  is  that  DoD  will  be 
living  with  major  systems  for  longer  periods 
of  time.  As  systems  get  older,  the  sustainment 
costs  usually  rise.  To  address  concerns  within 
these  two  areas,  the  Defense  Systems 
Affordability  Council,  chaired  by  the  Under 
Secretary  of  Defense  for  Acquisition  and  Tech¬ 
nology,  recently  directed  the  establishment  of 


ambitious  top  level  goals  for  reducing  total 
ownership  cost. 

Regarding  the  first  area  of  reducing  initial 
acquisition  costs,  an  SBA  process  increases  the 
likelihood  that  a  program  will  stay  within  its 
cost  objectives  during  the  acquisition  phase. 
The  synthetic  environment  reduces  the  reli¬ 
ance  on  costly  physical  prototypes  and  tests 
for  making  programmatic  decisions.  Physical 
tests  are  primarily  done  to  validate  models  and 
generate  confidence  in  the  use  of  the  models. 
These  validated  models  can  then  be  reused  to 
perform  numerous  design  simulations,  at  a 
fraction  of  the  cost  of  one  physical  test.  Simu¬ 
lations  also  help  to  focus  the  test  effort  on  the 
critical  evaluation  areas,  thereby  avoiding  un¬ 
necessary  physical  tests.  As  noted  by  Air  Force 
Major  General  Leslie  Kenne,  Program  Man¬ 
ager  for  the  Joint  Strike  Fighter  aircraft,  “We 
haven’t  begun  to  tap  all  the  benefits  that  mod¬ 
eling  and  simulation  has  to  offer  to  reduce  our 
testing  requirements.”^ 

Regarding  the  second  area  of  reducing  sus¬ 
tainment  costs,  an  SBA  process  increases  the 
likelihood  that  a  program  can  reduce  the 
annual  costs  of  operating  and  sustaining  sys¬ 
tems  until  and  including  disposal.  Although 
it’s  difficult  to  find  a  definitive  source  for  the 
data,  it  is  frequently  mentioned  that  approxi¬ 
mately  70  percent  of  the  total  costs  of  a  sys¬ 
tem  are  determined  by  the  time  a  new  pro¬ 
gram  receives  approval  to  start  (referred  to  as 
“Milestone  I”),  and  85  percent  are  determined 
by  the  time  a  program’s  design  is  selected 
(referred  to  as  “Milestone  11”).^  These  early 
decisions,  which  directly  affect  total  ownership 
cost,  are  currently  being  made  with  limited 
knowledge  of  system  cost,  schedule  and  per¬ 
formance  implications.  It  is  speculative  at  best 
to  determine  how  much  of  a  system’s  total 
ownership  cost  can  be  influenced  by  better 
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design  decisions.  Certainly,  however,  as  the 
ability  to  simulate  the  entire  system's  life  cycle 
continues  to  improve,  so  too  will  the  ability  to 
make  intelligent  tradeoffs  on  how  much  to 
spend  now  in  improvements  in  manufactur¬ 
ability,  reliability,  maintainability,  and  sup- 
portability  in  order  to  save  later.  A  significant 
value  of  SBA  is  that  it  allows  program  offices 
to  begin  evaluating  the  long-term  cost  impacts 
of  design  decisions  as  part  of  the  design 
process,  rather  than  relying  on  engineering 
change  proposals  and  modifications  to  fix  the 
problem  after  the  design  is  frozen. 

Guide  To  This  Report 

Chapter  Two,  Definitions  and  Terminology, 
lays  the  foundation  for  the  SBA  Vision 
Statement  and  Definition  that  follow. 

Chapter  Three,  Background,  presents  current 
policy  and  guidance,  previous  studies  and  con¬ 
clusions,  and  some  assumptions  and  trends 
upon  which  SBA  depends  upon. 

Chapter  Four,  Essential  Aspects  of  SBA, 
introduces  the  theoretical  practices  that 
support  an  SBA  process. 


Chapter  Five,  Expanding  the  SBA  Envelope, 
identifies  the  forces  that  will  generate  pro¬ 
gress  towards  implementing  SBA.  These 
include  the  forces  internal  to  a  program  that 
will  push  SBA  along,  as  well  as  the  exter¬ 
nal  pull  from  the  warfighting  and  resource 
allocation  communities. 

Chapter  Six,  A  Future  State  of  SBA,  develops 
a  model  and  vision  for  what  SBA  can  hope  to 
achieve  in  the  future,  by  identifying  notional 
SBA  components  and  their  interactions. 

Chapter  Seven,  Challenges  to  Implementing 
SBA,  describes  some  of  the  challenges  that  a 
program  may  face  as  it  tries  to  implement  an 
SBA  approach. 

Chapter  Eight,  Conclusions  and  Recommen¬ 
dations,  summarizes  our  conclusions  about  the 
prospects  for  implementing  SBA,  and  provides 
our  recommendations  for  making  the  transi¬ 
tion  to  this  new  way  of  acquiring  defense 
systems. 

What  we  present  throughout  this  report  we  have 
heard  many  times  from  many  people.  It  is  our 
hope  that  we  have  presented  our  research  in 
such  a  way  as  their  message  comes  across  loud 
and  clear  and  that  you  are  convinced  that  SBA 
is  a  better  way  of  doing  business. 
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DEFINITIONS  AND 
TERMINOLOGY 


Many  within  the  DoD,  the  Armed  Forces,  and 
industry  realize  the  enormous  potential  of  a 
Simulation  Based  Acquisition  process,  and 
SB  A  has  recently  been  the  topic  of  several  con¬ 
ferences  and  workshops.  There  has  been  some 
disagreement  and  confusion,  however,  in  get¬ 
ting  to  a  common  understanding  of  exactly 
what  SBA  is.  Certainly  SBA  is  a  new  way  of 
doing  business  for  acquiring  DoD  weapon  sys¬ 
tems,  which  implies  a  need  to  change  our  pro¬ 
cesses.  And  it  has  been  described  by  the  policy 
and  cultural  changes  necessary  to  bring  about 
this  changed  process,  the  favorable  environ¬ 
ment  necessary  to  speed  it  along,  as  well  as 
the  technical  impediments  to  its  swift  enact¬ 
ment.  But  what  exactly  is  SBA? 

M&S  Definitions 

First  we  need  to  clarify  some  terminology  used 
throughout  this  book.  (We  highly  recommend 
referring  to  DoD  5000.59-M,  DoD  Modeling 
and  Simulation  (M&S)  Glossary,  for  current 
definitions  in  this  fast  changing  area.  It  is  avail¬ 
able  on-line  on  the  DMSO  World  Wide  Web 
Home  page,  mentioned  previously  in  the  Pref¬ 
ace.)  According  to  DoD  5000.59-M,  a  model 
is  ‘‘a  physical,  mathematical,  or  otherwise 


logical  representation  of  a  system,  entity,  phe¬ 
nomenon,  or  process.”^  (Entities  are  “a  dis¬ 
tinguishable  person,  place,  unit,  thing,  event, 
or  concept  about  which  information  is  kept.”^) 
It  is  important  to  note  that  a  model  can  exist 
without  a  single  piece  of  software.^  It  can  be  a 
hardware  mockup  or  a  simple  equation  on  a 
piece  of  paper.  A  simulation  is  “a  method  for 
implementing  a  model  over  time.”^  Thus,  to 
tie  the  two  together,  simulations  are  the  soft¬ 
ware  that  implements  models  over  time,  within 
the  context  of  a  scenario.^ 

A  great  deal  of  confusion  often  results  from 
the  common  practice  of  using  the  terms  “mod¬ 
eling”  and  “simulation”  interchangeably,  as 
well  as  that  of  using  the  term  “M&S”  to  stand 
for  both  models  and  simulations  and  modeling 
and  simulation.  ^  In  this  report  we  use  the  term 
“M&S”  to  stand  for  modeling  and  simulation, 
which  is  an  analytical  problem-solving  ap¬ 
proach.^  Modeling  and  simulation,  as  defined 
in  the  DoD  M&S  Glossary,  is  “the  use  of  mod¬ 
els,  including  emulators,  prototypes,  simula¬ 
tors,  and  stimulators,  either  statically  or  over 
time,  to  develop  data  as  a  basis  for  making 
managerial  or  technical  decisions.”^ 
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Hierarchies  and  Classes  of  Models 
and  Simulations 

As  shown  in  Figure  2-1,  models  and  simula¬ 
tions  have  been  classified  hierarchically 
according  to  their  level  of  aggregation.  Aggre¬ 
gation  is  the  ability  to  group  entities,  while 
preserving  the  effects  of  entity  behavior  and 
interaction  when  grouped.^  The  four  general 
classifications  are  Engineering,  Engagement, 
Mission/Battle,  and  Theater/Campaign.^^ 
Engineering  models  and  simulations  are  at  the 
least  level  of  aggregation,  whereas  Theater/ 
Campaign  models  and  simulations  are  at  the 
highest.  Within  these  classifications,  models 
and  simulations  can  vary  in  their  level  of  detail 
or  representation, 

•  Engineering  level  models  and  simulations 
provide  measures  of  performance  (MOP) 
concerning  such  issues  as  design,  cost, 
manufacturing,  and  supportability.  They 


can  include  aerodynamics,  fluid  flow, 
acoustics,  and  fatigue,  as  well  as  physics- 
based  models  of  components,  subsystems, 
and  systems.” 

•  Engagement  level  models  and  simulations 
are  used  for  evaluating  the  effectiveness  of 
an  individual  system  against  another  sys¬ 
tem  in  one-on-one,  few-on-few,  and  many- 
on-many  scenarios.  They  provide  measures 
of  effectiveness  (MOE)  at  the  system-on- 
system  level.^^ 

•  Mission/Battle  level  models  and  simula¬ 
tions  are  used  for  evaluating  the  effective¬ 
ness  of  a  force  package,  or  multiple  plat¬ 
forms  performing  a  specific  mission.  They 
provide  MOE  at  the  force-on-force  level.^^ 

•  Theater/Campaign  level  models  and  simu¬ 
lations  are  used  to  evaluate  the  outcomes 
of  joint  and  combined  forces  in  a  theater 


Figure  2-1 .  Hierarchies  of  Models  and  Simulations 
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or  campaign  level  conflict.  They  provide 
measures  of  value  (sometimes  referred  to 
as  measures  of  outcome-MOO)  at  the 
highest  levels  of  conflict.^"^ 

This  classification  of  models  and  simulations 
is  significant  to  acquisition  programs,  because 
the  type  of  information  required  determines 
the  level  of  aggregation  to  be  used.  Solutions 
to  broad  issues,  such  as  mission  need  state¬ 
ments,  can  be  explored  with  highly  aggregated 
models  and  simulations.  More  focused  issues, 
such  as  the  maturing  of  the  design,  dictate  that 
the  models  and  simulations  move  towards  and 
into  the  engineering  category.  Programs  must, 
therefore,  move  up  and  down  this  ladder  of 
abstraction  to  tailor  the  models  and  simula¬ 
tions  to  their  needs.  For  example,  highly  ag¬ 
gregated  simulations  can  explore  the  battle¬ 
field  effects  of  increasing  an  aircraft’s  combat 
radius.  If  the  results  are  promising,  the  design 
team  could  then  use  engineering  level  models 


and  simulations  to  address  the  associated 
detailed  design  issues,  such  as  increasing  the 
aircraft’s  internal  fuel  capacity.  There  are  also 
additional  considerations  for  tailoring  the 
level  of  aggregation,  such  as  unnecessary 
computational  burden  and  the  complexity  of 
information  management. 

Models  and  simulations  have  long  been  clas¬ 
sified  into  the  three  classes  of  Live,  Virtual, 
and  Constructive,  which  attempt  to  delineate 
the  degrees  of  human  and  equipment  realism, 
as  shown  in  the  matrix  in  Figure  22}^  Live 
simulations  denotes  real  people  operating  real 
systems;  virtual  simulations  denotes  real 
people  operating  simulated  systems;  con¬ 
structive  models  and  simulations  denote  simu¬ 
lated  people  operating  simulated  systems. 
Smart  simulations  denotes  simulated  people 
operating  real  equipment.  The  live,  virtual  and 
smart  classes  are  applied  to  simulations,  but 
not  models  since  live  models  would  be  humans 


People 


Simulated 


Equipment 


Real 


Real  Simulated 


Figure  2-2.  Matrix  of  Classes  of  Models  and  Simulations 
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and  actual  equipment.  The  term  virtual  model 
would  be  misleading  since  virtual  simulations 
inject  humans-in-the-loop  to  exercise  motor 
control,  decision  making,  or  communications 
skills,  and  the  human  element  of  a  virtual  simu¬ 
lation  is  not  modeled  (the  simulated  systems 
in  virtual  simulations  would  be  made  up  of 
constructive  models).  On  the  other  hand,  in 
constructive  simulations,  real  people  stimulate 
(make  inputs)  to  the  constructive  models,  but 
the  people  are  not  involved  in  determining  the 
outcomes.  Hence  it  is  appropriate  to  have 
both  constructive  models  and  constructive 
simulations. 

As  noted  by  the  DoD  M&S  Glossary,  the  clas¬ 
sification  of  live,  virtual,  and  constructive  can 
be  somewhat  problematic  for  two  reasons. 
First,  there  is  no  clear  delineation  between 
the  categories,  because  the  degrees  of 
human  involvement  and  equipment  realism 


are  infinitely  variable.  The  second  reason  it  is 
a  problematic  classification  is  highlighted  by 
the  fact  that  the  bottom  right  quadrant  of 
Figure  2.2  has  not  been  previously  named 
to  indicate  the  class  of  simulated  people 
operating  real  equipment.  We  have  named 
this  class  of  simulations  as  smart  simulations. 

There  is  value,  however,  in  marrying  the  hier¬ 
archies  and  classes  of  models  and  simulations 
together,  as  shown  in  Figure  2-3,  to  show  the 
range  of  possibilities  when  discussing  M&S 
support  to  acquisition.  Constructive  models 
and  simulations  can  range  from  highly  aggre¬ 
gated,  theater  level  models  and  simulations, 
to  physics-based,  engineering  level  models  and 
simulations.  This  range  of  aggregation  can  be 
applied  to  virtual  simulations,  and  we  can 
envision  the  same  for  the  obverse  of  virtual 
simulations  (the  “smart”  classification).  The 
possible  combinations  are  wide-ranging,  in¬ 
cluding  such  hybrid  combinations  as  hardware/ 


Figure  2-3.  Hierarchies  and  Classes  of  Models  and  Simulations 
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software  in  the  loop  (HW/SWIL),  human  in 
the  loop  (HITL),  and  wargaming.  It  is  not 
necessary  to  package  every  model  or  simula¬ 
tion  into  a  particular  category — the  utility  is 
in  exploring  the  numerous  possible  hybrid 
combinations  available  and  tailoring  them  to 
the  requirement. 

While  the  classification  of  live  simulation  was 
useful  for  the  training  community,  the  concept 
of  real  people  and  real  equipment  takes  on  a 
different  purpose  for  the  acquisition  environ¬ 
ment.  In  the  training  vernacular  of  “all  but  war 
is  simulation,”  real  people  and  real  equipment 
are  still  simulations  of  the  real  event  of  war. 
For  the  acquisition  of  systems,  there  are  two 
roles  for  the  classification  of  real  people  and 
real  equipment.  The  first  role  is  testing  that 
which  cannot  be  adequately  depicted  in  the 
synthetic  environment,  because  of  insufficient 
knowledge  or  technology.  The  data  obtained 
from  these  tests  could  be  used  for  future  model 
development.  The  second  role  is  to  validate 
the  models  and  resultant  simulations.  The 
“live”  events  in  these  two  roles  can  range  from 
component  level  tests  to  large-scale  physical 
prototypes,  but  in  each  case  their  purpose  is 
to  validate.  The  first  role  validates  either  a  con¬ 
cept  or  design,  whereas  the  second  role  vali¬ 
dates  a  model  or  simulation. 

Virtual  and  Synthetic 

When  discussing  M&S  and  SBA,  it  is  impor¬ 
tant  to  have  a  common  understanding  of  the 
word  “virtual.”  In  the  most  general  sense,  vir¬ 
tual  “refers  to  the  essence  or  effect  of  some¬ 
thing,  not  the  fact,”^^  In  practice,  the  word  vir¬ 
tual  is  usually  paired  with  another  word,  as  in 
virtual  battlespace,  virtual  reality,  and  virtual 
prototype.  Virtual  battlespace  is  defined  as 
“the  illusion  resulting  from  simulating  the 
actual  battlespace.”^®  Virtual  reality  is  “the 


effect  created  by  generating  an  environment 
that  does  not  exist  in  the  real  world.  Usually, 
[virtual  reality  is]  a  stereoscopic  display  and 
computer-generated  three-dimensional  envi¬ 
ronment  which  has  the  effect  of  immersing  the 
user  in  that  environment.  This  is  called  the 
immersion  effect.  The  environment  is  inter¬ 
active,  allowing  the  participant  to  look  and 
navigate  about  the  environment,  enhancing 
the  immersion  effect.  Virtual  environment  and 
virtual  world  are  synonyms  for  virtual  real- 
ity.”^^  Synthetic  environments,  on  the  other 
hand,  are  defined  as  “...  simulations  that  rep¬ 
resent  activities  at  a  high  level  of  realism,  fi:om 
simulations  of  theaters  of  war  to  factories  and 
manufacturing  processes.  These  environments 
may  be  created  within  a  single  computer  or  a 
vast  distributed  network  connected  by  local 
and  wide  area  networks  and  augmented  by 
super-realistic  special  effects  and  accurate 
behavioral  models.  They  allow  visualization  of 
and  immersion  into  the  environment  being 
simulated.”^^  Finally,  virtual  prototypes  are  “a 
model  or  simulation  of  a  system  placed  in  a 
synthetic  environment,  and  are  used  to  inves¬ 
tigate  and  evaluate  requirements,  concepts, 
system  design,  testing,  production,  and  sustain¬ 
ment  of  the  system  throughout  its  life  cycle.”^ 

Life  Cycle  Cost  and  Total  Ownership  Cost 

The  concepts  of  life  cycle  cost  (LCC)  and  total 
ownership  cost  (TOC)  also  figure  prominently 
when  discussing  SBA.  The  term  TOC  often 
appears  to  be  replacing  the  term  LCC.  TOC 
is  the  totality  of  costs  associated  with  the 
Department  of  Defense,  including  the  costs  re¬ 
lated  to  weapon  systems,  whereas  LCC  is  the 
totality  of  costs  over  time  related  to  develop¬ 
ing,  acquiring,  operating,  supporting  and 
disposing  of  weapon  systems,^  The  Secretary 
of  Defense,  William  S.  Cohen,  states  in  his 
Annual  Report  to  the  President  and  the  Congress 
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that  “Total  ownership  cost  is  the  sum  of  all 
financial  resources  necessary  to  organize, 
equip,  and  sustain  military  forces  sufficient  to 
meet  national  goals  in  compliance  with  all 
laws;  all  policies  applicable  to  DoD;  all 
standards  in  effect  for  readiness,  safety,  and 
quality  of  life;  and  all  other  official  measures 
of  performance  for  DoD  and  its  compo¬ 
nents.”^  When  viewed  this  way,  LCC  inher¬ 
ently  refers  to  that  subset  of  TOC  that  has  to 
do  with  weapons  systems. 

In  practice,  however,  TOC  is  often  used  in  a 
weapon  system  context,  in  which  case  TOC 
and  LCC  appear  to  be  synonymous  terms,  as 
they  both  cover  all  costs  to  research,  develop, 
acquire,  own,  operate  and  dispose  of  a  weapon 
system  for  a  specific  (or  assumed)  number  of 
years.  However,  TOC  as  applied  to  weapon 
systems  seems  to  imply  a  greater  effort  to  cap¬ 
ture  more  of  the  costs  covered  by  both  defini¬ 
tions  than  has  heretofore  been  practiced 
routinely  in  DoD  under  the  banner  of  LCC.^ 
For  example,  if  a  new  system  or  concept  will 
require  additional  security  forces  as  part  of  its 
operation,  then  the  life  cycle  cost  of  this  sup¬ 
port  would  most  likely  be  factored  in  to  the 
TOC  of  the  system  or  concept,  but  would  prob¬ 
ably  not  have  been  included  under  LCC.  There 
are  many  on-going  debates  on  this  subject, 
which  we  don’t  intend  to  resolve  here.  In  this 
book  we  use  the  more-encompassing  term  TOC. 

SBA  Vision  Statement 

For  many  years,  the  training  community  has 
leveraged  the  strengths  of  M&S  to  augment 
live  training.  As  noted  by  General  Richard 
Hawley,  Commander  of  Air  Combat  Com¬ 
mand,  “...[M&S]  has  been  a  key  part  of  our 
training  for  many  years,  but. .  .we  Ve  never  fully 
exploited  the  contributions  that  modeling  and 
simulation  can  make  to  our  readiness 


programs.”^^  Similarly,  although  the  acquisi¬ 
tion  community  has  also  made  M&S  a  key  part 
of  the  systems  acquisition  process  for  many 
years,  it  too  is  beginning  to  realize  that  the 
benefits  of  M&S  haven’t  been  fully  exploited. 
In  his  16  March  1998  Memorandum,  the  Hon¬ 
orable  Jacques  S.  Gansler,  Under  Secretary 
of  Defense  (Acquisition  and  Technology), 
stated,  “As  we  undergo  changes  in  our  defense 
acquisition  infrastructure,  let  me  take  this 
opportunity  to  firmly  state  my  commitment  to 
the  use  of  M&S  in  the  acquisition  of  our  weap¬ 
ons  systems....  Therefore,  it  is  essential  that 
we  plan  for  the  use  of  M&S  in  our  acquisition 
strategies.  I  expect  programs  to  make  the 
upfront  investment  in  M&S  application  and 
technology  and  will  be  looking  for  evidence 
of  that  investment  in  program  planning  and 
execution.”^  Dr.  Gansler  goes  on  to  note  ad¬ 
ditional  steps  he  is  endorsing  for  the  SBA 
initiative  to  capitalize  on  the  current  efforts 
in  M&S. 

The  DoD,  in  collaboration  with  industry, 
developed  a  vision  statement  for  SBA  that 
envisions  an  acquisition  process  in  which  DoD 
and  industry  are  enabled  by  robust,  collabo¬ 
rative  use  of  simulation  technology  that  is 
integrated  across  acquisition  phases  and 
programs.  The  goals  of  SBA  are  to: 

1.  Substantially  reduce  the  time,  resources, 
and  risk  associated  with  the  entire  acqui¬ 
sition  process; 

2.  Increase  the  quality,  military  worth,  and 
supportability  of  fielded  systems,  while 
reducing  total  ownership  costs  throughout 
the  total  life  cycle;  and 

3.  Enable  Integrated  Product  and  Process 
Development  (IPPD)  across  the  entire 
acquisition  life  cycle.^^ 
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SBA  Definition 

This  vision  statement  promotes  discus¬ 
sions  about  SBA,  and  most  people  basically 
agree  with  its  intent.  However,  an  expanded 
definition  of  SBA  is  instructive: 

Simulation  Based  Acquisition  is  an 
iterative,  integrated  product  and  pro¬ 
cess  approach  to  acquisition,  using 
modeling  and  simulation,  that  enables 
the  warfighting,  resource  allocation, 
and  acquisition  communities  to  fulfill 
the  warfighter’s  materiel  needs,  while 
maintaining  Cost  As  an  Independent 
Variable  (CAIV)  over  the  system’s 
entire  life  cycle  and  within  the  DoD’s 
system  of  systems. 

A  discussion  of  each  of  the  definition’s  critical 
elements  follows. 

“Simulation  Based  Acquisition  is  an  iterative, 
integrated  product  and  process  approach  to 
acquisition”:  SBA  enables  Integrated  Product 
and  Process  Development  (IPPD)  teams  to 
converge  on  optimal  solutions  by  balancing 
requirements  through  an  iterative  design  pro¬ 
cess.  DoD  and  contractor  organizations  work 
internally  and  with  each  other  as  an  integrated 
team  effort.^  IPPD  used  to  mean  that  we  put 
a  logistician  on  the  design  team  to  translate 
his  maintenance  and  support  knowledge  into 
meaningful  design  guidelines.  Now  IPPD 
means  arming  the  hands-on  experts  with  the 
tools  to  rapidly  see  the  results  of  their  design 
inputs,  thereby  making  them  a  part  of  the 
design  team.  For  example,  using  three-dimen¬ 
sional  “solid”  models,  the  F-22  aircraft  pro¬ 
gram  enabled  two  mechanics  to  inject  main¬ 
tainability  changes  into  the  design  tradeoffs, 
because  they  were  able  to  visualize  the  system 
much  earlier  in  the  process. 


“...through  modeling  and  simulation”:  M&S 
activities  make  SBA  possible.  Stepping  into 
the  synthetic  environment  enables  exercising 
the  power  of  simulation.  Within  the  same  time 
frame,  many  more  analytical  excursions  can 
be  made  with  virtual  designs  than  would  be 
possible  with  physical  prototypes.  The 
increased  level  of  user  involvement,  coupled 
with  compressed  time  and  space  feedback, 
lead  to  better  learning  and  problem  solving. 
This  is  far  superior  to  the  traditional  approach, 
which  is  driven  by  real  experiences  using  physi¬ 
cal  prototypes.^^  M&S  facilitates  the  team  and 
increases  communication,  making  team 
members  more  effective. 

“...the  warfighting,  resource  allocation,  and 
acquisition  communities”:  SBA  is  more  than 
just  acquisition.  SBA  helps  to  link  the  DoD’s 
three  principal  decision  support  systems:  the 
Requirements  Generation  System,  the  Plan¬ 
ning  Programming  and  Budgeting  System,  and 
the  Acquisition  Management  System.^^  Figure 
2-4  shows  the  relationship  of  these  three  com¬ 
munities.  The  name  of  the  community  repre¬ 
senting  the  Requirements  Generation  System 
is  changed  to  the  more  expansive  term  of 
warfighting  community,  which  includes:  re¬ 
quirements  personnel,  operators,  maintainers/ 
sustainers,  and  trainers.  The  resource  alloca¬ 
tion  community  is  also  an  integral  part  of  the 
acquisition  process,  as  it  allocates  the  program 
budgets  (subject  to  approval  by  OSD  and  Con¬ 
gress),  and  has  the  burden  of  balancing  the 
Services’  and  DoD’s  budget(s).  The  acquisi¬ 
tion  community,  as  used  here,  includes  both 
government  and  industry  agencies  involved  in 
developing  and  fielding  military  systems. 

“...to  fulfill  the  warfighter’s  materiel  needs 
while  maintaining  Cost  As  an  Independent 
Variable  (CAIV)”:  indicates  the  use  of  a  strat¬ 
egy  that  balances  mission  needs  with  projected 
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Figure  2-4.  Three  Principal  Interacting  Communities 


out-year  resources.  We  are  now  saying  that  we 
will  trade  performance  in  order  to  achieve  our 
cost  objective.  By  linking  improved  cost 
models  with  our  computer-aided  engineering 
tools,  we’ll  be  able  to  better  predict  the  costs 
of  different  alternatives  so  as  to  make  better 
informed  tradeoff  analyses. 

“...over  the  system’s  entire  life  cycle”  means 
to  look  both  within  and  across  all  phases  of 
the  program,  as  early  as  possible  during  the 
acquisition  of  the  system.  It  encompasses  the 
collaborative  use  of  simulation  beyond  the 
traditional  performance  issues  to  address  the 
system’s  entire  life  cycle  cost  issues  during  the 
design,  to  include  manufacturability,  support- 
ability,  lethality,  sustainability,  mobility, 
survivability,  flexibility,  interoperability, 
reliability,  and  affordability. 


“...and  within  the  DoD’s  system  of  systems.” 
This  signifies  to  fully  explore  the  system’s 
interaction  within  and  impact  upon  the  DoD’s 
system  of  systems,  to  capture  the  desire  for 
effective  total  systems  integration,  as  well  as 
the  collaborative  use  of  M&S  across  programs. 
As  the  United  States  participates  in  greater 
numbers  of  combined  operations,  this  aspect 
of  the  definition  will  have  to  be  expanded  to 
include  a  “system  of  systems”  look  across  allied 
systems  and  programs  as  well. 

Figure  2-5  graphically  illustrates  the  three 
dimensions  that  SBA  attempts  to  integrate. 
First,  the  vertical  dimension  is  where  M&S  has 
been  traditionally  applied  within  each  program 
phase,  without  much  regard  to  reuse  later  in 
the  program.  The  second  dimension  SBA 
attempts  to  integrate  is  the  horizontal  appli¬ 
cation  of  M&S  across  the  phases  of  the 
program.  More  than  just  sticking  with  and 
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Figure  2-5.  Application  of  M&S  Across  Three  Dinnensions 


growing  models  over  the  life  of  a  program,  this 
means  addressing  the  entire  life  cycle’s  issues 
as  early  as  possible  during  the  design.  The  ac¬ 
quisition  community  must  stay  focused  on  the 
real  goal,  which  is  to  produce  superior  weapon 
systems,  not  superior  virtual  prototypes.  The 
key  to  success  is  to  get  the  design  right  before 
building  the  system,  after  which,  in  effect,  the 
design  becomes  frozen.  This  horizontal  appli¬ 
cation  across  phases  of  the  program  indicates 
an  attempt  to  explore  the  cost  drivers  across 
the  entire  life  c^cle  of  the  system,  and  when  it 
makes  sense,  to  address  those  issues  in  the 
design.  For  example,  more  reliable  systems  can 
translate  into  better  combat  effectiveness,  as 
well  as  potential  manpower  decreases  for 
maintenance,  decreased  mobility  footprint, 
and  reduced  sparing  levels.  Acquiring  good 
cost  figures  for  these  types  of  operations  and 
support  costs  across  the  expected  life  of  the 


system  will  help  determine  how  much  should 
be  spent  on  reliability  during  system  design. 
After  production,  as  we  continue  to  refine  our 
system  models  with  field  data  and  operator 
input,  the  simulations  will  help  us  decide  if  we 
need  to  modify  or  build  a  new  system,  as  well 
as  consider  non-materiel  solutions.  Lx)oking 
at  the  furthest  point  in  the  life  cycle,  there  may 
even  be  disposal  costs  that  could  be  mitigated 
by  changes  in  the  design.  While  the  importance 
of  looking  at  disposal  costs  may  seem  some¬ 
what  far-fetched,  it  is  interesting  to  note  that 
the  Army  spends  about  $100  million  a  year 
demilitarizing  ammunition,  of  which  $13 
million  alone  is  for  stocks  from  World  War  1.^^ 
If  we  don’t  start  looking  at  disposal  costs  dur¬ 
ing  the  design,  we  may  find  that  in  the  future, 
we  won’t  be  able  to  afford  to  buy  the  next- 
generation  system,  because  we  couldn’t  afford 
to  dispose  of  the  current  one. 


Finally,  the  third  dimension  SBA  attempts  to 
integrate  is  across  programs.  If  we  continue 
to  maximize  each  system  alone,  the  overall  sys¬ 
tem  of  systems  will  be  less  than  optimal.  Pro¬ 
grams  need  to  recognize  the  value  of  the 
interaction  of  their  system  within  the  overall 
system  of  systems.  Individual  programs  can¬ 
not  afford  to  build  everything  themselves;  they 
need  to  rely  on  each  other  for  their  similar 
needs.  There  are  few  design  tradeoff  analyses 
being  conducted  within  the  services  between 
their  own  programs,  much  less  between 
different  services’  programs.  Currently,  the 
Joint  Requirements  Oversight  Council 
(JROC)  reviews  all  Mission  Need  Statements 
(MNS)  for  joint  applicability,  and  follows  up 
on  major  programs  before  milestone  reviews. 
To  do  these  tradeoffs  effectively,  however,  they 
have  to  be  made  at  a  level  much  lower  than 
the  JROC.  The  capability  to  do  these  tradeoff 
analyses  will  begin  to  extend  beyond  the 
traditional  interface  control  procedure 
method,  which  allows  each  individual  system 


to  be  optimized,  but  which  results  in  the  over¬ 
all  system  of  systems’  performance  being  sub¬ 
optimized.  Failure  to  look  at  the  big  picture 
can  also  result  in  over-designing  systems  as 
well  as  unnecessary  duplication  of  effort,  both 
of  which  waste  resources. 

When  we  begin  to  look  outside  a  single  sys¬ 
tem,  we  see  even  fewer  interactions  of  these 
issues  in  the  context  between  systems,  as  for 
example  the  maintenance  and  logistics  con¬ 
siderations  between  the  next  generation 
amphibious  assault  vehicle  and  the  ships  that 
will  transport  it.  We  see  this  happening  even 
today  within  large,  integrated  weapon  systems 
such  as  submarines  and  aircraft,  where  the 
major  sub-components  (sonars,  torpedoes, 
missiles,  and  munitions,  etc.)  are  designed  and 
developed  independently  and  integrated  later 
after  each  has  been  built.  The  design  inter¬ 
face  is  controlled  by  an  interface  control  docu¬ 
ment  (ICD),  with  little  to  no  concurrent  design 
tradeoffs  possible  across  the  ICD. 
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BACKGROUND 


There  are  numerous  studies,  papers  and  pro¬ 
gram  examples  that  highlight  the  benefits 
M&S  can  provide  in  technical  risk  reduction, 
time  savings,  direct  cost  savings  and/or  cost 
avoidance,  and  management  risk  reduction. 
Since  many  of  these  documents  are  continu¬ 
ally  being  refined  (particularly  in  the  dynamic 
growth  area  of  M&S),  it  is  important  to  view 
them  first-hand  for  possible  changes.^  For 
example,  the  Army  recently  coined  the  phrase 
Simulation  and  Modeling  for  Acquisition, 
Requirements  and  Training,  or  SMART,  to 
describe  their  initiative  for  SBA.^ 

This  chapter  presents  current  policy  and  guid¬ 
ance,  previous  studies  and  conclusions,  and 
some  assumptions  and  trends  upon  which  SBA 
depends.  Thus  far,  the  only  reference  to  Simu¬ 
lation  Based  Acquisition  that  appears  in  the 
Defense  Acquisition  Deskbook  is  Dr. 
Gansler's  16  March  1998  memorandum  pre¬ 
viously  mentioned  in  Chapter  Two.  We  expect 
this  will  soon  change,  however,  as  the  acquisi¬ 
tion  community  continues  to  implement  this 
“smarter”  way  of  doing  business.  The  follow¬ 
ing  synopses  are  a  quick  walk-through  of 
recent  information  relevant  to  SBA. 


Guidance 

DoD  5000.1,  ^^Defense  Acquisition^^ 

This  directive  encourages  the  use  of  M&S  by 
stating  that  “models  and  simulations  shall  be 
used  to  reduce  the  time,  resources,  and  risks 
of  the  acquisition  process  and  to  increase  the 
quality  of  the  systems  being  acquired.  Repre¬ 
sentations  of  proposed  systems  (virtual  pro¬ 
totypes)  shall  be  embedded  in  realistic, 
synthetic  environments  to  support  the  various 
phases  of  the  acquisition  process,  from 
requirements  determination  and  initial 
concept  exploration  to  the  manufacturing  and 
testing  of  new  systems,  and  related  training.”^ 

DoD  5000.2-R,  ^Mandatory  Procedures  for 
MDAPs  and  MAIS  Acquisition  Programs^^ 

This  directive  mandates  that  “accredited  mod¬ 
eling  and  simulation  shall  be  applied,  as 
appropriate,  throughout  the  system  life  cycle 
in  support  of  the  various  acquisition  activities: 
requirements  definition;  program  manage¬ 
ment;  design  and  engineering;  efficient  test 
planning;  result  prediction;  and  to  supplement 
actual  test  and  evaluation;  manufacturing;  and 
logistics  support.  [Program  Managers]  shall 
integrate  the  use  of  modeling  and  simulation 
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within  program  planning  activities,  plan  for  life 
cycle  application,  support,  and  reuse  models 
and  simulations,  and  integrate  modeling  and 
simulation  across  the  functional  disciplines. 

Regarding  test  and  evaluation,  it  further  states 
that  “modeling  and  simulation  shall  be  an 
integral  part  of  test  and  evaluation  plan¬ 
ning... Test  and  evaluation  programs  shall  be 
structured  to  integrate  all  developmental  test 
and  evaluation  (DT&E),  operational  test  and 
evaluation  (OT&E),  live-fire  test  and  evalua¬ 
tion  (LFT&E),  and  modeling  and  simulation 
activities  conducted  by  different  agencies  as 
an  efficient  continuum.  Ail  such  activities  shall 
be  part  of  a  strategy  to  provide  information 
regarding  risk  and  risk  mitigation,  to  provide 
empirical  data  to  validate  models  and  simula¬ 
tions,  to  permit  an  assessment  of  the  attainment 
of  technical  performance  specifications  and  sys¬ 
tem  maturity,  and  to  determine  whether  systems 
are  operationally  effective,  suitable,  and 
survivable  for  intended  use.” 

For  operational  test  and  evaluation,  it  states 
that  “The  use  of  modeling  and  simulation  shall 
be  considered  during  test  planning.  Whenever 
possible,  an  operational  assessment  shall  draw 
upon  test  results  with  the  actual  system,  or  sub¬ 
system,  or  key  components  thereof,  or  with 
operationally  meaningful  surrogates.  When 
actual  testing  is  not  possible  to  support  an 
operational  assessment,  such  assessments  may 
rely  upon  computer  modeling,  simulations 
(preferably  with  real  operators  in  the  loop), 
or  an  analysis  of  information  contained  in  key 
program  documents.  However,  as  a  condition 
for  proceeding  beyond  TRIP,  initial  opera¬ 
tional  test  and  evaluation  shall  not  comprise 
an  operational  assessment  based  exclusively 
on  computer  modeling;  simulation;  or,  an 
analysis  of  system  requirements,  engineering 
proposals,  design  specifications,  or  any  other 


information  contained  in  program  documents 
(10  USC2399).  The  extent  of  modeling  and 
simulation  usage  in  conjunction  with  opera¬ 
tional  and  test  evaluation  shall  be  explained 
in  the  Test  and  Evaluation  Master  Plan....”’ 

For  Live  Fire  Test  and  Evaluation  on  other 
than  ACAT  lA  programs,  it  states:  “...Alter¬ 
natively,  in  the  case  of  a  covered  system  (or 
covered  product  improvement  program  for  a 
covered  system),  the  USD(A&T)  or  the  CAE 
[Component  Acquisition  Executive]  may 
waive  the  application  of  the  required  surviv¬ 
ability  and  lethality  tests  and  instead  allow  test¬ 
ing  of  a  system  or  program  by  firing  munitions 
likely  to  be  encountered  in  combat  at  compo¬ 
nents,  subsystems,  and  subassemblies,  together 
with  performing  design  analyses,  modeling  and 
simulation,  and  analysis  of  combat  data  in  lieu 
of  testing  the  complete  system  configured  for 
combat.”* 

Regarding  the  four  key  systems  engineering 
tasks  of  requirements  analysis,  functional 
analysis/allocation,  design  synthesis  and  verifi¬ 
cation,  and  system  analysis  and  control,  it  states 
that  “The  verification  of  the  design  shall  in¬ 
clude  a  cost-effective  combination  of  design 
analysis,  design  modeling  and  simulation,  and 
demonstration  and  testing. . . .  ”^ 

DoD  Directive  5000.59,  ^DoD  Modeling  and 
Simulation  (M&S)  Managemenf’ 

The  purpose  of  this  directive  is  threefold.  First, 
it  establishes  DoD  policy,  assigns  responsi¬ 
bilities,  and  prescribes  procedures  for  the  man¬ 
agement  of  M&S.  Second,  it  establishes  the 
DoD  Executive  Council  for  Modeling  and 
Simulations  (EXCIMS).  Third,  it  establishes 
the  Defense  Modeling  and  Simulation  Office 
(DMSO). 
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It  directs  that  it  is  DoD  policy  that  “Invest¬ 
ments  shall  promote  the  enhancements  of 
DoD  M&S  technologies  in  support  of  opera¬ 
tional  needs  and  the  acquisition  process; 
develop  common  tools,  methodologies,  and 
databases;  and  establish  standards  and  proto¬ 
cols  promoting  the  internetting  data  exchange, 
open  system  architecture,  and  software  reus¬ 
ability  of  M&S  applications.  Those  standards 
shall  be  consistent  with  current  national,  Fed¬ 
eral,  DoD -wide  and,  where  practicable,  inter¬ 
national  standards  for  open  systems...  The 
DoD  Components  shall  establish  verification, 
validation,  and  accreditation  (WA)  policies 
and  procedures  for  M&S  applications  man¬ 
aged  by  the  DoD  Component.  The  ‘DoD 
M&S  Executive  Agent’  shall  establish  WA 
procedures  for  that  application....  M&S 
applications  used  to  support  the  major  DoD 
decision  making  organizations  and  processes 
(such  as  the  Defense  Planning  and  Resources 
Board;  the  Defense  Acquisition  Board;  the 
Joint  Requirements  Oversight  Council;  and 
the  DoD  Planning,  Programming,  and  Bud¬ 
geting  system)...  shall  be  accredited  for  that 
use  by  the  DoD  Component  for  its  own  forces 
and  capabilities.  Each  DoD  Component  shall 
be  the  final  authority  for  validating  and  ac¬ 
crediting  representations  of  its  own  forces  and 
capabilities  in  joint  and  common  use  M&S. 
Each  Component  shall  be  responsive  to  the 
other  Components  to  ensure  that  its  forces  and 
capabilities  are  appropriately  represented  in 
the  development  of  joint  and  common  use 

M&S.”'o 

DoD  5000.59-P,  ^^Modeling  and  Simulation 
(M&S)  Master  Plan^^ 

This  plan  establishes  numerous  M&S  activi¬ 
ties.  First,  it  establishes  the  DoD  vision  for 
DoD  M&S  and  outlines  a  strategy  for  achiev¬ 
ing  future  DoD  M&S-based  capabilities. 


Second,  it  assigns  M&S  implementation  re¬ 
sponsibilities  and  provides  guidelines  for  the 
development,  cooperation,  and  coordination 
of  DoD  M&S  efforts.  Third,  it  establishes 
DoD  M&S  objectives,  identifies  action,  and, 
where  possible,  assigns  responsibilities  for  ac¬ 
complishing  them.  Fourth,  it  provides  a  basis 
for  developing  supporting  plans  and  programs, 
including  the  DoD  Modeling  and  Simulation 
Investment  Plan  (MSIP),  and  the  DoD 
Component’s  M&S  master  and  investment 
plans.  Fifth,  it  provides  justification  for  re¬ 
source  allocations  to  M&S  within  DoD  Com¬ 
ponent  programming  and  budgeting  processes 
and  fosters  the  integration  of  the  defense  and 
civilian  M&S  bases  into  a  unified  national  and 
international  base  using  common  standards, 
processes  and  methods.^^ 

DoD  Manual  5000.59-M,  “DoD  Modeling  and 
Simulation  (M&S)  Glossary^^^ 

This  manual  prescribes  a  uniform  glossary  of 
M&S  terminology  for  use  throughout  the  De¬ 
partment  of  Defense.  In  addition  to  the  main 
glossary  of  terms,  it  includes  a  list  of  M&S- 
related  abbreviations,  acronyms,  and  initials 
commonly  used  within  the  DoD. 

DoD  Instruction  5000.61,  ^DoD  Modeling  and 
Simulation  (M&S)  Verification^  Validation  and 
Accreditation  (W&A)*^ 

This  Instruction  implements  policy,  assigns 
responsibilities,  and,  prescribes  procedures 
for  the  W&A  of  DoD  M&S.  It  designates 
the  Defense  Modeling  and  Simulation 
Office  (DMSO)  as  the  DoD  W&A  focal 
point,  to  be  the  central  source  of  DoD  W^&A 
data  and  information  and  to  assist  DoD  and 
non-DoD  organizations,  as  requested,  in  re¬ 
solving  W&A  issues  or  obtaining  informa¬ 
tion  on  DoD  W&A  practices.  It  specifies  that 
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information  and  data  on  VV&A  activities  will 
be  readily  available  through  the  DoD  M&S 
Resource  Repository  (MSRR)  system  includ¬ 
ing,  as  a  minimum,  DoD  Component  VV&A 
policies  and  procedures,  V&V  results,  and 
accreditation  documentation.  This  instruction 
also  specifies  minimum  documentation 
requirements  for  verification  and  validation 
information,  and  accreditation  results. 

“Department  of  Defense  Verification^  Valida¬ 
tion  and  Accreditation  (W&A)  Recommended 
Practices  Guide** 

This  guide  provides  background  and  informa¬ 
tion  on  recommended  principles,  processes, 
and  techniques  for  use  in  DoD  W&A  efforts 
that  support  the  analysis,  acquisition,  and 
training  communities. 

“Simulation,  Test,  and  Evaluation  Process 
(STEP)  Guidelines** 

Provides  a  set  of  guidelines  for  the  program 
manager  to  refer  to  in  implementing  the 
DoD’s  October  3, 1995  direction  to  make  the 
Simulation,  Test  and  Evaluation  Process  an 
integral  part  of  Test  and  Evaluation  Master 
Plans.  STEP  proposes  a  Model-Simulate-Fix- 
Test-Iterate  approach,  and  is  defined  as  .  .an 
iterative  process  that  integrates  simulation  and 
test  for  the  purpose  of  interactively  evaluat¬ 
ing  improving  the  design,  performance,  joint 
military  worth,  survivability,  suitability,  and 
effectiveness  of  systems  to  be  acquired  and 
improving  how  those  systems  are  used.”  STEP 
is  described  as  “a  test  and  evaluation  answer 
to  the  DoD  challenges  of  implementing  IPPD 
and  SBA.”i5 


Previous  and  On-Going  Efforts 

“Study  on  the  Effectiveness  of  Modeling  and 
Simulation  in  the  Weapon  System  Acquisition 
Process,**  Science  Applications  International 
Corporation,  October  1996 

The  purpose  of  this  report  was  to  cite  docu¬ 
mented  contributions  to  the  total  acquisi¬ 
tion  process.  It  concluded  that  “There  is 
consistent  evidence  of  M&S  being  used  effec¬ 
tively  in  the  acquisition  process  but  not  in  an 
integrated  manner  across  programs  or  func¬ 
tions  within  the  acquisition  process.  Substan¬ 
tial  evidence  has  been  collected  from  individual 
success  stories,  though  the  benefits  are  not 
readily  quantifiable  into  a  general  standard.  The 
key  is  in  focusing  on  the  integration  of  M&S 
applications,  across  acquisition  programs  and 
throughout  the  process,  not  in  exploring  the 
applications  themselves. . 

In  attempting  to  quantify  metrics,  it  noted 
“cost  savings  are  especially  difficult  to  quan¬ 
tify”  and  are  often  “more  correctly  classified 
as  ‘cost  avoidance’  and  are  measures  of  sig¬ 
nificant  additional  work  or  results  that  were 
obtained  using  M&S  tools  which  would  have 
cost  the  reported  ’savings’  if  they  had  been 
obtained  by  more  traditional  methods.”  This 
study  grouped  the  challenges  that  preclude  the 
seamless  use  of  M&S  in  the  acquisition  pro¬ 
cess  into  technical,  cultural,  and  managerial 
challenges,  as  noted  below: 

Technical  Challenges: 

•  Interoperability  of  M&S  Tools; 

•  Availability  of  Data  Descriptions; 

•  Security/Sensitivity  of  Data; 

•  Physics-based  M&S; 

•  Hardware  and  Software  Limitations; 

•  Variable  Resolution. 
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Cultural  Challenges: 

•  Acquisition  Processes; 

•  Incentives  for  M&S  Use; 

•  M&S  Workforce; 

•  Acceptance  of  M&S. 

Managerial  Challenges: 

•  OSD  and  Service  Guidance; 

•  Ownership  of  Data; 

•  W&A  Requirements; 

•  Funding  Process; 

•  Use  of  System  Models. 

This  report  defined  SBA  as  a  “term  to 
characterize  the  general  approach  of  signifi¬ 
cantly  increased  use  of  M&S  tools  and  the  new 
processes  which  they  enable  in  a  new,  more 
integrated  approach  to  program  develop¬ 
ment”  (italics  emphasis  added).  It  further 
characterized  SBA  in  terms  of  a  systems 
engineering  process  by  saying  “The  positive 
results  of  simulation  efforts  in  systems  engi¬ 
neering  have  become  evident  in  what  we  refer 
to  as  Simulation  Based  Acquisition.” 

Finally,  Appendix  C  of  the  October  1996  study 
contains  a  useful  summary  of  previous  studies 
and  recommendations  dating  from  1989  to  1995. 

Common  Operating  Digital  Environment 
(CODE) 

CODE,  a  DoD  initiative  related  to  SBA,  is 
being  co-sponsored  by  the  Assistant  Deputy 
Undersecretary  of  Defense  for  Logistics 
Reinvention  and  Modernization  (ADUSD/ 
LR&M),  and  the  National  Defense  Industrial 
Association  (NDIA).  A  joint  industry-  and 
government-working  group  is  developing  a 
framework  for  a  “large-grained  interoperabil¬ 
ity”  technical  environment  to  enable  digital 
weapon  system  life  cycle  information  manage¬ 
ment  by  2002.  The  vision  is  for  government 


and  industry  partners  to  create,  exchange,  and 
sustain  information  solely  in  a  digital  environ¬ 
ment.  The  CODE  will  provide  access  to  digi¬ 
tal  data  across  the  virtual  extended  enterprise, 
and  the  functionality  to  support  business  pro¬ 
cesses  and  decision  making.  The  initial  report 
is  cognizant  of  SBA  and  notes  that  the  Joint 
Technical  Architecture  and  its  subsystem  High 
Level  Architecture  need  to  be  reviewed  with 
other  stovepipe  architectures  to  ensure  an 
optimum  life  cycle  environment.^^ 

Defense  Systems  Affordability  Council 
(DSAC) 

In  December  1997,  the  DSAC  identified  three 
fundamental  areas  contributing  to  the 
affordability  of  acquisition  programs:  1)  reduc¬ 
tion  of  total  ownership  costs;  2)  50  percent 
reduction  of  acquisition  program  cycle  time 
for  new  systems;  and  3)  realistic  programming 
and  program  stability  (zero  percent  program 
cost  growth)  enabled  by  a  broad  range  of 
potential  improvements  in  requirements  set¬ 
ting,  funding  management,  and  acquisition 
practices.  As  noted  in  the  previous  chapter, 
Simulation  Based  Acquisition  contributes  to 
all  three  of  these  objectives. 

Joint  SBA  Task  Force 

The  Acquisition  Council  of  the  Executive 
Council  on  Modeling  and  Simulation  commis¬ 
sioned  a  six  month  effort  to  develop  a  road 
map  in  which  DoD  and  Industry  are  enabled 
by  robust,  collaborative  use  of  simulation  tech¬ 
nology  that  is  integrated  across  acquisition 
phases  and  programs.  The  Terms  of  Reference 
for  this  effort  called  for  six  major  items  to  be 
included  in  the  road  map: 

1.  near-  and  long-term  DoD  actions  needed 
to  accelerate  the  SBA  concept; 
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2.  industry  actions  needed  to  accelerate 
SBA; 

3.  notional  representations  of  systems  archi¬ 
tectures  and  a  conceptual  framework  to 
identify  key  requirements  for  seamless 
data  transfer  and  interaction; 

4.  opportunities  for  reuse  and  commonality 
across  programs; 

5.  primary  ownership  of  each  module  in  the 
systems  architecture;  and 

6.  estimated  government  and  industry 
investments  needed  to  implement  the  pro¬ 
posed  road  map,  and  methods  to 
determine  the  return  on  investment. 


The  SBA  Task  Force  is  ongoing  during  the 
publication  of  this  research  effort,  and  their 
results  will  be  presented  at  a  conference  in 
November  1998. 

Assumptions,  llrends,  and  SBA  Enablers 

Brigadier  General  Robert  Armbruster, 
Deputy  for  Systems  Acquisition  at  the  U.S. 
Army  Aviation  and  Missile  Command,  notes 
that  three  change  factors  now  make  SBA  pos¬ 
sible:  technology,  advocacy,  and  Life  Cycle 
Management.^ 

Technology  has  advanced  to  the  point  where 
it  is  often  less  expensive  to  simulate  processes, 
components,  and  systems,  than  it  is  to  physi¬ 
cally  create  or  run  them.  Computing  resources 


Figure  3-1 .  Growth  in  Computing  Power 
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that  were  worth  $30K  in  1960  only  cost  about 
10  cents  today.^^  In  1965,  Gordon  Moore  (who 
co-founded  Intel  Corporation  in  1968)  made 
the  observation  that  computing  power  was 
doubling  every  18-24  months.  Now  known  as 
Moore’s  Law,  his  theory  has  proved  to  be  re¬ 
markably  accurate:  in  26  years,  the  number  of 
transistors  on  a  chip  has  increased  more  than 
3200  times,  from  2300  on  the  4004  processor 
in  1971,  to  7.5  million  on  the  Pentium  II  pro- 
cessor.^^  This  exponential  growth  in  comput¬ 
ing  power  is  shown  graphically  in  Figure  3-1.^^ 

In  1960,  many  companies  began  using  simula¬ 
tion  tools,  but  they  were  often  inadequate.^^ 
Recent  increases  in  computing  power  and 
simulation  tools,  however,  have  greatly 
increased  the  capabilities  of  M&S  efforts  to  a 
program.  While  physical  models  and  proto¬ 
types  do  not  represent  reality  completely, 
programs  have  grown  accustomed  to  using 
them  and  understand  their  limitations.  Now, 
programs  are  finding  that  in  many  instances 
the  computer  models  are  more  correct  than 
the  physical  models.  The  following  example 
highlights  this  point.  The  experiment,  to  be  run 
in  both  the  physical  and  virtual,  was  to  test  the 
ability  of  a  ship’s  compartment  to  withstand 
an  explosion.  During  the  validation  testing  of 
the  ship’s  model,  the  test  engineers  could  not 
get  the  computer  simulation  results  to  agree 
with  the  actual  physical  test  results.  The  test 
design  for  the  compartment  called  for  a  cir¬ 
cumference  weld,  which  had  been  correctly 
included  in  the  finite  element  computer 
model.  However,  as  the  physical  prototype  had 
been  incorrectly  spot  welded,  the  results  of  the 
physical  test  did  not  match  the  results  from 
the  simulation.  When  the  error  was  realized, 
it  was  easier  (and  more  cost-effective)  to 
change  the  computer  model  to  a  spot  weld, 
and  re-run  the  simulation,  at  which  time,  the 
results  matched.^  Another  example  is  from 


the  automotive  industry.  A  simulation  analy¬ 
sis  showed  that  the  vehicle  skin  would  rip 
below  a  certain  thickness.  Not  believing  the 
simulation,  the  engineers  overrode  the  analy¬ 
sis  of  the  simulation  and  made  the  vehicle  skin 
thinner.  During  testing  the  skin  ripped  as  the 
simulation  had  predicted.  It  was  at  this  point 
that  the  engineers  became  believers  in  the 
results  of  simulations. 

This  leads  to  the  second  change  factor  of 
“advocacy.”  Leadership  is  essential  to  effec¬ 
tive  change  in  an  organization  and  today 
change  within  an  organization  must  be  led  and 
not  just  managed.^^  Within  OSD  and  the  Ser¬ 
vices  senior  leadership  within  the  acquisition 
community  is  taking  an  active  role  towards  the 
implementation  of  SBA.  Advocacy  for  SBA, 
however,  is  not  solely  limited  to  senior  lead¬ 
ership  but  can  be  found  at  all  levels  of  the 
acquisition  community.  At  the  Electronic 
Proving  Ground  at  Fort  Huachuca,  Arizona, 
they  are  finding  that  simulations  can  be  bet¬ 
ter  than  physical  tests.  Security  issues  and 
civilian  communication  interference  concerns 
limit  what  can  be  physically  tested  on  some 
systems.  Testing  in  the  virtual  environment 
removes  such  barriers.  Thus  some  simulations 
can  be  more  realistic  than  what  can  actually 
be  tested  in  the  physical.^^  Success  stories,  such 
as  this  and  the  examples  in  the  preceding  para¬ 
graph,  are  beginning  to  make  believers  out  of 
the  most  hardened  skeptics  and  are  helping 
to  promote  this  technology. 

And  the  third  change  factor  contributing  to 
make  SBA  possible  is  the  desire  to  be  able  to 
perform  Life  Cycle  Management — ^simulation 
enables  us  to  do  LCM.^^  Simulation  makes  it 
possible  to  really  get  cost  on  the  table  and  on 
equal  footing  with  the  performance  require¬ 
ments.  This  forces  the  requirements  commu¬ 
nity  to  make  the  hard  tradeoffs  as  part  of  the 
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design  process.  As  noted  by  BG  Bergantz,  Pro¬ 
gram  Manager  for  the  Army’s  Comanche 


helicopter  program,  SBA  enables  us  to  “put 
the  intellectual  before  the  physical.”^^ 
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ESSENTIAL  ASPECTS  OF  SBA 


This  chapter  presents  five  essential  features 
of  a  Simulation  Based  Acquisition  process 
which,  when  implemented,  will  present  the  ac¬ 
quisition  community  with  capabilities  not  cur¬ 
rently  imbedded  in  the  current  process.  First, 
SBA  will,  through  early  and  continuous 
involvement,  allow  the  user  to  define,  refine, 
and  balance  requirements.  This  is  the  critical 
first  step  in  producing  better,  faster,  and 
cheaper  material  solutions.  Second,  an  SBA 
process  supported  by  the  synthetic  environ¬ 
ment,  allows  design  teams  to  concurrently  ex¬ 
plore  greater  numbers  of  possible  material 
solutions  than  is  possible  within  the  current 
acquisition  process.  Third,  the  iterative  nature 
of  the  SBA  design  process  will  enable  IPPD 
teams  to  converge  systematically  on  optimal 
solutions  more  efficiently  than  is  currently 
possible.  Fourth,  SBA  will  support  changing 
the  role  of  testing  into  that  of  being  an  inte¬ 
gral  part  of  the  design  process.  Finally,  SBA 
supports  informed  tradeoff  analyses  through 
a  Decision  Risk  Analysis  process.  This  chap¬ 
ter  will  explore  each  aspect  separately.  Though 
presented  individually,  when  the  five  essen¬ 
tial  aspects  of  SBA  are  combined,  their  syner¬ 
gistic  effect  towards  assisting  the  acquisition 
process  is  greater  than  the  sum  of  the 
individual  parts. 


Balancing  Requirements 

Early  user  involvement  is  essential  in  defin¬ 
ing,  refining,  and  balancing  requirements.  The 

following  quotations  emphasize  this  point: 

•  “The  best  way  to  define  a  program  is 
through  M&S — plunk  the  user  down  into 
the  virtual  world,  have  him  play  with  the 
concept,  and  work  out  the  tactics,  training, 
and  procedures — take  the  user  out  of  the 
abstract  and  into  the  virtual  combat 
world.”' 

•  “SBA  and  M&S  enable  you  to  play  optom¬ 
etrist  with  the  user — do  you  like  it  this  way, 
or  that?”2 

•  “We  need  more  user  input  during  devel¬ 
opment  of  the  system.  Simulation  brings 
the  warfighters  and  engineers  together  so 
that  each  has  a  better  appreciation  of  the 
future  system  and  its  interaction  in  the 
future  battlefield.”^ 

•  “Spiral  development  concentrates  on  sol¬ 
dier  feedback.  It  involves  the  soldier  who 
uses  the  equipment  directly  in  the  devel¬ 
opment  process,  brings  all  the  parties 
together — from  the  user  to  the  developer 
to  industry.”'' 


•  “The  warfighter  only  has  a  general  notion 
of  his  requirements  up  front,  and  is  unable 
to  give  a  detailed  description  early  on.  M&S 
enables  the  user  and  the  developer  to 
walk  up  the  spiral  development  ladder 
together.”^ 

The  traditional  acquisition  process  assumes 
the  user  completely  understands  the  require¬ 
ments  and  the  impact  each  has  on  the  pro¬ 
gram.  To  be  more  specific,  it  has  been  assumed 
that  the  user  completely  understands  the  over¬ 
all  cost  associated  with  each  requirement. 
According  to  Lieutenant  General  George 
Muellner,  Air  Force  Principal  Deputy  for 
Acquisition,  former  Director  of  Requirements 
for  Air  Combat  Command,  and  former  Direc¬ 
tor  of  the  Joint  Advanced  Strike  Technology 
Program:  “The  old  way  of  doing  business  was 
for  the  user  to  throw  his  requirement  over  the 
fence  to  the  acquisition  community,  who 
would  then  spend  five  years  working  the  mar¬ 
gins,  when  90%  of  the  performance  was 
already  locked  up.  Now  we’re  getting  where 
we  can  keep  the  trade  space  open,  don’t  lock 
up  the  requirements,  and  force  the  require¬ 
ments  people  to  deal  early  on  with  these 
issues.”^ 

Balancing  requirements  in  the  defense  acqui¬ 
sition  business  means  bringing  into  balance 
what  the  user  wants  with  what  can  be 
affordably  accomplished,  or  as  stated  by  Lt. 
Gen.  Muellner,  converting  the  warfighter’s 
wants  into  affordable  needs.^  According  to  Col 
Cuff,  U.S.  Army  Training  and  Doctrine  Com¬ 
mand  (TRADOC)  Systems  Manager  (TSM) 
for  the  Crusader  field  artillery  program:  “With 
simulation,  the  user  has  a  better  understand¬ 
ing  of  the  cause  and  effect  relationship  be¬ 
tween  requirements,  cost,  and  schedule.  The 
TSM  gets  to  see  the  operational  impact  of 
design  and  requirements  tradeoffs.  The  user 


always  wants  everything,  but  previously  did  not 
have  an  appreciation  for  the  cost  and  sched¬ 
ule  impacts  that  requirements  had  on  program 
execution.”^  An  SBA  process  supports  the  user 
through  visual  representation  of  the  various 
design  alternatives  early  in  the  design  phase. 
The  user  can  then  compare  alternatives  and 
provide  input  to  the  design  team  on  the  value 
of  competing  designs  with  relationship  to  his 
requirements.  Visualization  also  makes  iden¬ 
tification  and  resolution  of  many  issues  easier 
and  faster.  The  entire  IPPD  team  can  see 
where  the  issues  are  and  then  focus  on  how  to 
solve  the  problem. 

System  requirements  are  really  a  combination 
of  many  things:  the  user’s  operational  needs, 
logistics  concerns,  lessons  learned,  environ¬ 
mental  issues,  etc.  These  system  requirements 
must  then  be  balanced  in  terms  of  cost  and 
force  effectiveness.  The  user  plays  a  key  role 
in  balancing  requirements,  and  the  applied  use 
of  M&S  across  the  phases  of  a  program  can 
significantly  affect  the  outcome  of  the  final 
system. 

Figure  4-1  shows  the  JSF  program’s  depiction 
of  this  balancing  act  between  cost  and  capa¬ 
bility  in  determining  the  most  affordable 
solution.  The  JSF  sets  cost  objectives  to  bal¬ 
ance  mission  needs  with  projected  out-year 
resources,  taking  into  account  anticipated 
process  improvements  in  both  DoD  and  the 
defense  industries.^ 

Numerous  Design  Alternatives 

Concurrent  systems  engineering  can  be  viewed 
as  a  process  of  balancing  requirements,  or 
making  compromises  between  competing 
requirements.  Its  primary  function  is  to  ensure 
the  product  meets  the  customer’s  cost,  sched¬ 
ule,  and  performance  needs  encompassing  the 
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Affordability 


Figure  4-1 .  Affordability:  Capability  vs.  Cost 


total  product  life  cycle.^*^  This  systems  engi¬ 
neering  approach  is  not  new;  what  is  new  is 
the  ability  to  conduct  many  parts  of  this  pro¬ 
cess  in  the  synthetic  environment  using  M&S. 
The  benefit  of  doing  this  process  in  the  syn¬ 
thetic  environment  is  that  designers  and  de¬ 
velopers  can  try  things  without  fear  of  failure. 
A  recent  report  on  virtual  prototyping  noted 
that  during  the  design  cycle,  a  hesitancy  for 
revisions  exists  once  a  marginally  working  sys¬ 
tem  has  been  achieved,  as  a  result  of  the  in¬ 
creased  risk  associated  with  making  changes.” 
This  increased  risk  is  a  result  of  the  time  lost 
and  the  associated  cost  of  building  alternative 
hardware,  even  though  the  alternative  may  be 
a  superior  design. 

Because  making  design  changes  in  a  synthetic 
environment  can  be  faster  and  cheaper  (and 
therefore  less  risky)  than  making  changes  to  a 
physical  prototype,  an  IPPD  team  is  able  to 
explore  more  design  options.  Harvard 
University’s  Stefan  Thomke,  in  his  research 
with  the  automobile  industry,  cites  a  powerful 


example.  An  automotive  company  found  that 
an  assumption  regarding  a  component 
involved  in  side-impact  crashes  was  in  fact 
false.  This  assumption  had  never  been  tested 
because  it  seemed  obvious  that  making  a  com¬ 
ponent  in  a  car  stronger  would  improve  crash- 
worthiness.  Physical  prototypes  and  crashes 
were  reserved  for  high  payoff  issues.  Engineers 
were  not  willing  to  use  costly  physical  testing 
to  test  assumptions  they  were  confident  about. 
Because  it  was  quick  and  inexpensive  to  check 
out  the  assumption  using  simulation,  one 
engineer  insisted  on  it.  Surprisingly,  the  team 
discovered  that  strengthening  the  component 
actually  decreased  crashworthiness,  by  unex¬ 
pectedly  propagating  the  load  to  another  part 
of  the  vehicle.  The  solution  was  to  reduce  the 
stiffness  of  the  component,  which  went  against 
the  conventional  wisdom.  This  discovery  led 
to  a  reevaluation  of  other  reinforced  areas  and 
has  improved  the  crashworthiness  of  all  cars 
under  development.  These  and  other  design 
changes  improved  crashworthiness  by  30  per¬ 
cent — a  significant  increase.  Interestingly,  the 
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time  and  cost  to  build  physical  vehicle  proto¬ 
types  for  the  two  verification  experiments 
at  the  completion  of  the  project  exceeded 
the  time  and  cost  of  the  entire  advanced 
development  M&S  project.^^ 

Once  a  physical  prototype  exists  it  is  difficult 
for  people  to  envision  a  new  concept  that  is 
substantially  different.  It  is  very  compelling  for 
the  design  team  to  get  to  a  point  design,  and 
then  consider  possible  excursions  or  engineer¬ 
ing  changes  from  that  point  design,  rather  than 
examining  the  entire  design  space.  Designing 
in  the  synthetic  environment,  however, 
decreases  the  cost  and  time  of  looking  at 
alternative  designs. 

Thomke  notes  that  the  advantages  of  sub¬ 
stituting  real  physical  objects  with  virtual  ex¬ 
perimentation  can  be  very  substantial —  once 
set  up,  virtual  tests  can  be  run  at  very  little 


additional  cost  per  run.^^  In  addition,  he 
notes  several  other  benefits  of  virtual 
experimentation.  These  include: 

•  Better  depth  and  quality  of  the  analyses 
because  it  is  possible  to  slow  the  test  events 
down  and  zoom  in  on  minute  areas. 

•  Information  from  problem-solving  cycles 
becomes  available  to  other  development 
tasks  earlier  in  the  development  process, 
therefore  other  tasks  can  proceed  more 
quickly,  and  higher  degrees  of  design 
concurrence  are  feasible. 

•  With  faster  problem-solving  cycles,  design¬ 
ers  will  be  able  to  push  (or  “front-load”) 
problem-discoveiy  to  earlier  development 
phases  and  thus  make  problem-related 
engineering  changes  earlier  and  at  a  lower 
cost. 


Simulation  Based  Acquisition  Process 


INTERACTIVE 

QFD  ANALYSIS  DIGITAL  PHYSICAL  TEST 


DISTRIBUTED  INTERACTIVE  ENVIRONMENT 


Figure  4-2.  SBA  Process 
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•  Reduced  cost  and  time  to  get  experi¬ 
mental  feedback  encourages  the  design 
team  to  conduct  more  diverse  “what  if 
experiments  which,  in  turn,  may  lead  to 
the  discovery  of  more  effective  (and 
unanticipated)  design  changes.^^ 

The  Iterative  Design  Process 

The  Joint  Strike  Fighter  (JSF)  program  por¬ 
trays  the  process  of  iteratively  designing  a 
system  using  the  SBA  process  as  shown  in 
Figure  4-2.^^  The  JSF  process  is  a  building 
block  approach,  with  each  step  building  upon 
the  previous  steps  as  well  as  iterating  within 
the  steps  and  between  steps.  It  begins  with  the 
Delphi  Process,  which  is  a  method  devised  by 
the  RAND  Corporation  to  obtain  expert  judg¬ 
ment  from  multiple  experts  without  many  of 
the  biases  associated  with  interpersonal  pres¬ 
sures.^^  These  results  are  then  fed  into  a  Qual¬ 
ity  Functional  Deployment  (QFD)  analysis, 
which  refines  the  requirements  into  ways  to 
accomplish  those  requirements.  QFD  provides 
a  way  of  tracking  and  tracing  tradeoff  analy¬ 
ses  through  various  levels,  from  requirements 
through  design  decisions  to  production  and 
support  processes,  while  preserving  traceabil¬ 
ity  to  user  needs. These  results  are  then  used 
to  build  the  appropriate  level  of  constructive 
simulation.  At  first,  these  simulations  will  be 
at  an  aggregated  campaign  level,  but  will  con¬ 
tinue  to  be  iterated  and  refined  back  through 
the  previous  two  steps  and  down  the  construc¬ 
tive  simulation  hierarchy  to  mission/battle, 
engagement,  and  finally  the  engineering  level. 
As  more  is  learned,  the  simulation 
progresses  to  interactive  digital  simulation 
and  interaction  within  a  virtual  environ¬ 
ment,  which  introduces  the  human-in-the- 
loop.  Again,  there  can  be  much  iteration 
within  each  step,  as  well  as  feedback  to  ear¬ 
lier  steps  to  capture  the  knowledge  gained. 


As  the  program  matures  into  flight  test, 
these  results  are  also  fed  back  into  the  pro¬ 
gram  to  capture  the  knowledge,  and  to  help 
challenge  and  possibly  revise  the  inputs  to  the 
process. 

Once  a  program  reaches  the  constructive 
simulation  step,  it  has  the  option  to  begin 
interacting  with  other  systems  in  a  distributed 
interactive  environment.  This  can  be  a  pow¬ 
erful  tool  for  the  IPPD  team  because  it  enables 
them  to  receive  feedback  on  their  system  well 
before  the  design  is  locked  in.  Virtual  systems 
can  participate  in  exercises  and  experiments 
to  determine  their  effect  on  the  battlefield,  and 
those  results  and  warfighter  feedback  are  cap¬ 
tured  early  enough  in  the  program  to  help  the 
design  team  conduct  more  informed  tradeoff 
analyses.  In  effect,  the  warfighter  becomes  an 
integral  part  of  the  design  team. 

Converging  on  an  Optimal  Design 

Another  view  of  this  systems  engineering  and 
simulation  process  was  developed  by  United 
Defense  Limited  Partnership  and  is  being  used 
on  the  Crusader  field  artillery  system.  Their 
Simulate,  Emulate,  Stimulate  process  is  shown 
in  Figure  4-3  as  a  top-down  view,  and  in  Figure 
4-4  from  a  side  view.^^ 

The  Simulate,  Emulate,  Stimulate  process 
starts  with  a  low  fidelity  model,  and  moves 
through  the  four  steps  of  analyze,  design, 
evaluate,  and  revise,  successively  converging 
until  reaching  a  high  fidelity  model  of  the  final 
system.  Early  models  of  the  system  are  used 
in  a  series  of  simulations  to  evaluate  concepts 
of  potentially  high-risk  areas  to  identify  and 
resolve  deficiencies.  As  the  system  design 
matures,  some  subsystem  simulations  are 
replaced  with  emulations,  which  are  models 
that  accept  the  same  inputs  and  produce  the 


4-5 


Simulate  -  Emulate  -  Stimulate  Process 


Figure  4-4.  Stimulate-Emulate-Stimulate  Process,  Side  View 


same  outputs  as  a  given  system.^^  These  emu¬ 
lations  have  a  higher  level  of  fidelity  and  trans¬ 
late  to  a  more  detailed  evaluation  of  the  sys¬ 
tem  performance.  Further  design  maturation 
allows  the  model  to  stimulate  actual  system 
hardware,  leading  to  the  final  stage  of  bench 
top  integration  and  then  system  test.^ 

SBA  is  about  using  simulation  to  explore  the 
design  space,  to  validate  designs,  and  to  verify 
that  the  proposed  design  will  meet  the  end- 
user’s  expectations  and  is  manufacturable, 
supportable,  and  affordable.  Simulation 
enables  the  team  to  see  the  results  of  their 
design  decisions  not  only  in  terms  of  perfor¬ 
mance,  but  also  to  project  the  cost  of  those 
decisions  across  the  system’s  life  cycle.  The 
rapid  feedback  from  the  design  iterations  helps 
the  team  converge  on  an  optimal  solution — 
otherwise  they’ll  continue  to  work  on  the  prob¬ 
lem  until  they  run  out  of  time.  It’s  impossible 
to  know  if  a  single  design  is  the  best  possible 
solution;  however  it  is  possible  to  evaluate  one 
design  against  another.  The  confidence  in 
reaching  an  optimal  design  should  increase  as 
the  number  of  designs  explored  increases,  par¬ 
ticularly  if  there  is  significant  feedback  and 
learning  taking  place  between  the  design 
iterations. 

Data  and  personnel  interfaces  need  to  be 
seamless  and  integrated  to  facilitate  rapid 
design  iterations.  This  requires  an  integrated 
data  environment  to  work  effectively.  A  Prod¬ 
uct  Information  Management  (PIM)  tool  is 
extremely  important  to  manage  across  the  vir¬ 
tual  enterprise  and  can  facilitate  widely  sepa¬ 
rated  teams.  It  controls  and  provides  easy 
access  to  the  large  quantities  of  engineering 
and  product-related  data  that  are  generated 
during  concurrent  engineering,  while  tracking 
the  numerous  rapid  changes  from  different 
sources  in  the  organization  that  often  occur 


at  the  same  time.^^  Concurrent  engineering 
demands  a  great  deal  of  work-in-process 
information.  The  PIM  links  the  engineering 
tools  with  the  other  tools  necessary  to  manage 
a  fully  integrated  environment,  such  as  office 
automation,  systems  engineering,  business 
management,  configuration  management, 
reference  library,  and  particularly  web  and 
Internet  connectivity  for  cost-effective 
subcontractor  access. 

The  integrated  data  environment  is  changing 
the  entire  design  process.  For  example,  on  the 
Crusader  program,  critical  information  no 
longer  exists  in  specifications  and  drawings. 
It’s  now  in  very  large  databases  that  need  to 
be  managed  and  linked.  Requirements  for  the 
Crusader  now  exist  as  a  highly  structured, 
hierarchical,  linked  data  set  of  over  7,000 
requirements.  Using  their  integrated  data 
environment  management  system,  the  Cru¬ 
sader  design  team  can  quickly  identify  the  col¬ 
lateral  impact  of  a  requirements  change.^^  The 
team  can  trace  the  top-level  warfighting  tasks 
all  the  way  down  to  the  detailed  design  and 
back  up  to  the  final  product.  During  design 
iterations,  if  the  final  product  falls  short  of 
expectations  in  some  areas  (for  example  in 
initial  acquisition  cost,  interoperability,  or 
maintainability),  this  traceability  from  require¬ 
ments  to  design  will  be  key  to  enabling  the 
program  office,  in  conjunction  with  the 
warfighter,  to  make  smart  tradeoffs. 

Test  as  an  Integral  Part  of  Design 

The  traditional  “build  -  test  -  fix”  or  “test  - 
fix  -  test”  process  is  giving  way  to  a  new  pro¬ 
cess  in  which  test  is  becoming  an  integral  part 
of  design.  As  mentioned  before  in  Chapter 
Three,  the  Simulation,  Test,  and  Evaluation 
Process  (STEP)  Guidelines  calls  it  a  “model  - 
simulate  -  fix  test  -  iterate”  approach. 
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wherein  there  are  many  iterative  loops 
possible  in  this  process.^'^ 

Dr.  Henry  Dubin,  Technical  Director  of  the 
Army's  Operational  Test  and  Evaluation  Com¬ 
mand  (OPTEC),  emphasizes  this  changing 
process  by  saying  that  we  no  longer  “conduct 
test  and  evaluation  for  the  purpose  of  deter¬ 
mining  pass  or  fail  of  critical  threshold 
parameters... [we  now]  focus  testing  on  what 
the  acquisition  team  needs  to  learn — test  to 
learn,  and  evaluate  to  understand  the  [opera¬ 
tional]  impact  of  what  we  have  learned  on 
mission  accomplishment."^ 

When  viewing  the  test  process,  it's  important 
to  keep  in  mind  that  both  physical  tests  and 
virtual  tests  have  limitations.  Dr.  John  Foulkes, 
Director  of  the  Army's  Test  and  Evaluation 
Management  Agency,  notes  that  the  “test  and 
evaluation  community  should  view  anything 
short  of  actual  war  as  being  simulation,  includ¬ 
ing  live  testing.  Program  Managers  should 
ensure  the  right  mix  of  constructive  (digital) 
simulation,  virtual  simulation,  and  live  simu¬ 
lation  (testing),  and  continually  re-evaluate 
why  and  how  we  test.  Constructive  and  virtual 
simulation  should  reduce  the  burden  on  the 
amount  of  live  simulation  required,  thereby 
reducing  the  time  and  cost  of  testing.”^^ 

Models,  by  definition,  do  not  represent  real¬ 
ity  completely. This  applies  to  both  physical 
and  synthetic  models.  Models  are,  therefore, 
an  attempt  at  replicating  reality.  Many  reasons 
may  prevent  a  model  from  accurately  repre¬ 
senting  reality  or  being  a  high  fidelity  model. 
Inaccuracy  of  the  model  can  be  the  result  of 
the  inability  to  capture  all  the  attributes  of  the 
real  situation,  possibly  because  of  human 
error,  ignorance,  time  limits,  or  economics.^ 
Many  times  we  use  M&S  to  overcome  physi¬ 
cal  limitations  of  live  testing,  including  safety 


concerns,  interoperability  issues,  interference 
with  civilian  activities,  non-availability  of  the 
threat,  and  affordability.^^  In  part  M&S  is  done 
to  reduce  the  cost  of  representing  aspects  of 
the  real  that  are  irrelevant  to  the  experiment, 
as  well  as  to  control  out  some  aspects  of  the 
real  to  simplify  the  analysis  of  the  results.^ 

Dr.  Dubin  proposes  a  test  strategy  of  using 
M&S  to  design  and  develop  the  system,  to 
mature  the  models  with  the  system,  and  to  use 
the  test  data  for  calibrating  and  maturing  the 
system  model.  By  truly  integrating  testing  early 
on  we  should  gain  synergies  to  greatly  enhance 
the  overall  value  of  the  program.  He  empha¬ 
sized  his  point  by  stating  the  converse:  “Build¬ 
ing  a  simulation  solely  for  the  purpose  of 
reducing  your  Milestone  III  testing  require¬ 
ments  is  almost  always  a  waste  of  money.”^^  Dr. 
Phil  Coyle,  Director  of  Operational  Test  and 
Evaluation,  Office  of  the  Secretary  of  Defense, 
noted  the  same  importance  of  early  test  involve¬ 
ment:  “M&S  and  testing  are  intertwined;  when 
they  are  not,  neither  is  effective.”^^ 

The  benefits  of  M&S  to  the  test  and  evalua¬ 
tion  community  are  numerous,  among  them 
the  ability  to: 

•  conduct  full  and  continuous  evaluation  of 
the  system; 

•  evaluate  system  performance  where  live 
simulation  is  neither  feasible  or  practical; 

•  identify  and  concentrate  the  live  simulation 
resources  in  the  high  risk  areas; 

•  stress  the  system  at  less  than  system  level; 

•  conduct  excursions  for  the  development  of 
test  and  evaluation  plans  and  to  identify  live 
simulation  scenarios.^^ 


Involving  the  tester  early  in  a  program  may 
also  allow  the  tester  to  influence  the  design  to 
minimize  what  must  be  tested  and  to  facili¬ 
tate  the  testing  that  must  be  conducted.  The 
knowledge  gained  through  M&S  of  a  system 
will  enable  the  test  community  to  design  and 
conduct  more  effective  and  efficient  testing. 

The  Crusader  program  noted  that  the  biggest 
challenge  is  to  combine  testing  and  M&S  to 
leverage  savings  in  time  and  resources,  while 
satisfying  the  decisionmakers  that  the  require¬ 
ments  are  met.^  The  test  community  needs  to 
be  on  board  early  as  part  of  the  design  solution. 
This  way  they  gain  credibility  in  the  models  as 
they’re  being  used  early  in  the  program,  and 
build  in  confidence  in  the  models  through 
incremental  testing. 

The  intent  of  modeling  and  testing  is  also 
beginning  to  change.  For  example,  the  role  of 
the  Navy’s  David  Taylor  Model  Basin  has 
become  more  of  validating  models  rather  than 
designing  models.  The  purpose  of  testing  is 
evolving  into  validating  the  model  so  that  the 
analytical  calculations  derived  from  the  model 
are  believable. 

A  mix  of  analytical  and  physical  testing  is  often 
the  right  answer,  depending  upon  the  cred¬ 
ibility  and  confidence  in  the  tools.  New  pro¬ 
gram  starts  may  require  physical  prototypes 
to  prove  out  the  models,  since  there  may  not 
be  known  models  from  which  to  build.  But  the 
goal  remains  the  same — to  conduct  a  combi¬ 
nation  of  M&S  and  testing  in  order  to  get  a 
better  product.  As  one  tester  stated:  “The  old 
paradigm  was  to  test  a  company  of  tanks.  The 
new  paradigm  will  be  to  test  a  platoon,  but  to 
simulate  a  battalion.”^^ 


Decision  Risk  Analysis 

Though  modeling  and  simulation  costs  are 
coming  down,  they  still  can  be  very  expensive 
to  apply.  Programs  probably  cannot  afford  to 
use  M&S  on  every  issue  and  need  the  ability 
to  prioritize  their  efforts.  One  way  to  priori¬ 
tize  efforts  is  through  a  methodology  called 
Decision  Risk  Analysis  (DRA),  which  quanti¬ 
fies  and  ties  together  cost,  schedule,  perfor¬ 
mance,  producibility,  risk,  and  quality  to 
permit  informed  tradeoff  analyses.^^  A  deci¬ 
sion  risk  analysis  tool  quantitatively  provides 
the  program  manager  with  a  means  of  assess¬ 
ing  a  program’s  probability  of  achieving 
program  success,  while  employing  a  CAIV 
strategy.  If  the  assessed  risk  is  low  and  well 
known,  combinations  of  many  program  activi¬ 
ties  may  be  acceptable.  If  assessed  risk  is  sub¬ 
stantial  and/or  unknown,  a  detailed  break  out 
of  activities  accompanied  with  detailed  time, 
cost,  and  performance  estimates  for  comple¬ 
tion  of  those  activities  from  assessment  in  a 
Delphi-type  environment  may  be  required. 

One  such  tool  is  the  Venture  Evaluation  and 
Review  Technique  (VERT),  which  is  a  govern¬ 
ment-tool  designed  to  do  DRA.^’  It  is  a  Monte 
Carlo  simulation-based  tool  that  aids  program 
managers  in  measuring  risk  in  an  unbiased 
manner.  VERT  uses  the  same  approach  as  a 
Gannt  chart  and  it  adds  probability  to  make  it 
a  risk  measurement  tool.  To  build  a  more 
accurate  Gannt  chart  you  interview  IPT  mem¬ 
bers  to  get  a  more  accurate  assessment  of  how 
long  activities  will  take.  You  end  up  with  a 
range  of  time  an  event  could  take,  along  with 
a  corresponding  probability.  You  can  then  run 
a  series  of  “what  ifs”  to  determine  various  out¬ 
comes  as  parameters  are  changed.  The  Gannt 
chart  will  allow  you  to  identify  the  program’s 
critical  path  for  cost,  schedule,  and  perfor¬ 
mance.  You  can  then  focus  simulation  in  those 
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high-risk  areas  to  reduce  programmatic  risk. 
If  the  risk  is  low,  no  modeling  may  be  required, 
or  a  low  fidelity,  kinematics-based  CAD/CAM 
model  may  be  all  that  is  required  for  proof  of 
principle.  On  the  other  hand  if  the  risk  is 
high,  a  higher  fidelity  simulation  or  physi¬ 
cal  test  may  be  required  for  proof  of  concept. 


Implementing  a  program  like  VERT  can  take 
a  lot  of  manpower  to  collect  and  maintain  the 
data;  however,  it  can  pay  big  dividends  by  help¬ 
ing  determine  the  high  risk  areas.  A  DRA 
approach  helps  the  manager  determine  where 
to  employ  modeling  and  simulation  efforts  and 
to  what  level  of  fidelity. 


4-10 


ENDNOTES 


1.  Will  Brooks,  Chief,  Simulation  Branch,  Army 
Materiel  Systems  Analysis  Activity,  interview  by 
authors,  11  February  1998. 

2.  Colonel  Lynn  Carroll,  USAF,  Warfighter  Train¬ 
ing  Research  Division  Chief,  Air  Force  Research 
Laboratory,  briefing  to  the  Joint  Strike  Fighter 
Mission  Level  Strategy  session,  30  March  1998. 

3.  Rob  Beadling,  John  J.  McMullen  Associates,  Inc., 
interview  by  authors,  6  March  1998. 

4.  LTG  Kern,  USA,  “SBA:  Fielding  a  Capability 
Based  Force,”  presentation  and  remarks  at  Army 
SBA  Conference,  Orlando,  FL,  21-22  January 
1998. 

5.  Colonel  Picanso,  USAF,  Electronic  Systems  Cen¬ 
ter,  comments  during  presentation  at  Joint  Strike 
Fighter  Mission  Level  Strategy  session,  30  March 
1998. 

6.  LTG  George  Muellner,  USAF,  Air  Force  Princi¬ 
pal  Deputy  for  Acquisition,  interview  by  authors, 
30  March  1998. 

7.  Colonel  Kenneth  Konwin,  USAF,  Director  of 
Defense  Modeling  and  Simulation  Office, 
interview  by  authors,  31  July  1998. 

8.  Colonel  Michael  Cuff,  U.S.  Army,  TRADOC 
Systems  Manager  for  Cannon  Systems,  interview 
by  authors,  2  April  1998. 

9.  Major  General  Leslie  J.  Kenne,  Program  Man¬ 
ager,  Joint  Strike  Fighter  Program,  ‘‘Joint  Strike 
Fighter— The  Affordable  Solution”  30  April  1998 
briefing,  on-line  reference  www.jast.mil/html/ 
progbriefs.htm  on  22  April  1998 

10.  Defense  Acquisition  Deskbook,  Information 
Structure  2.6,1  Systems  Engineering,  Frontline 
Wisdom  and  Advise,  AFMC  -  The  Engineering 
Process  From  Industry  Point  of  View,  Submitted 
by  Mr.  Kammer,  HQ  AFMC/ENPP  (09  June 
1998). 


11.  Tim  Thornton,  Coleman  Research  Company 
(CRC),  Collaborative  Virtual  Prototyping  System 
(CVPS),  9  January  1997. 

12.  Stefan  Thomke,  “Simulation,  Learning  and  R&D 
Performance:  Evidence  from  Automotive 
Development,”  Research  Policy  27  (1)  (May 
1998),  55-74. 

13.  Ibid, 

14.  Ibid. 

15.  Colonel  Phil  Faye,  USAF,  Joint  Strike  Fighter 
Program  briefing,  30  March  1998. 

16.  For  more  information  on  Delphi,  see  Defense 
Acquisition  Deskbook,  Information  Structure 
1.2.4  Establish  Risk  Management  Strategy, 
AFMC — Identify  the  Ranking  of  Risk,  Submit¬ 
ted  by  Major  Paul  Loughnane,  USAF,  AFMC/ 
ENPI  (December  1997) 

17.  For  more  information  on  QFD,  see  the  DoD 
Guide  to  Integrated  Product  and  Process  Develop¬ 
ment,  Office  of  the  Undersecretary  of  Defense 
(Acquisition  and  Technology),  5  February  1996. 

18.  Bruce  Van  Wyk,  United  Defense  Limited  Part¬ 
nership,  Crusader  Program  briefing,  2  April  1998. 

19.  DoD  5000.59-M,  DoD  Modeling  and  Simulation 
(M&S)  Glossary,  December  1997. 

20.  Bruce  Van  Wyk,  United  Defense  Limited  Part¬ 
nership,  Crusader  Program  briefing,  2  April  1998. 

21.  John  Krouse,  “Manager’s  Guide  to  Computer 
Aided  Engineering,”  pp.  156-157. 

22.  Bruce  Van  Wyk,  United  Defense  Limited  Part¬ 
nership,  Crusader  Program  briefing,  2  April  1998. 


4-11 


23.  Dr.  John  B.  Foulkes;  Director,  U.S.  Army  Test 
and  Evaluation  Management  Agency;  Presenta¬ 
tion  and  remarks  at  Army  SBA  Conference,  21- 
22  January  1998. 

24.  Dr.  Philip  E.  Coyle  and  Dr.  Patricia  Sanders, 
Simulation,  Test,  and  Evaluation  Process:  STEP 
Guidelines,  4  December  1997,  pp.  1-3. 

25.  Dr.  Hank  Dubin;  Technical  Director,  U.S.  Army 
Operational  Test  and  Evaluation  Command;  Pre¬ 
sentation  and  remarks  at  Army  SBA  Conference, 
21-22  January  1998. 

26.  Foulkes,  ibid. 

27.  Thomke,  ibid. 

28.  Ibid. 

29.  Dubin,  ibid. 

30.  Thomke,  ibid. 

31.  Dubin,  ibid. 

32.  Dr.  Philip  E.  Coyle;  Director,  Operational  Test 
&  Evaluation,  Office  of  the  Secretary  of  Defense; 
Presentation  to  1997  Defense  Modeling  and 
Simulation  Office  Industiy  Days,  22  May  1997. 


33.  Foulkes,  ibid. 

34.  Colonel  William  Sheaves,  U.S.  Army;  Project 
Manager,  Crusader  field  artilleiy  program;  Pre¬ 
sentation  and  remarks  at  Army  SBA  Conference 
(21-22  January  1998). 

35.  John  Haug;  Director,  Technical  Mission,  Test  and 
Evaluation  Command,  U.S.  Army,  interview  by 
authors,  11  February  1998. 

36.  Dr.  Stuart  W.  Olson,  STRICOM,  AMSTI-E, 
Briefing  to  MG  John  E.  Longhouser,  Com¬ 
mander,  US  Army  Test  and  Evaluation  Com¬ 
mand  and  BG  John  P.  Geis,  Commander,  US 
Army  Simulation,  Training  and  Instrumentation 
Command,  7  February  1997. 

37.  The  U.S.  Army  Logistics  Management  College, 
Fort  Lee,  VA  teach  courses  on  VERT.  For  more 
information  see  on-line  http:/Avww.afanc.army.mil/ 


4-12 


5 

EXPANDING 
THE  SBA  ENVELOPE 


In  this  chapter  we  describe  how  Simulation 
Based  Acquisition  practices  within  the  Depart¬ 
ment  of  Defense  may  be  expanded.  To  set  the 
stage,  we  present  the  factors  influencing  the 
extent  to  which  a  program  can  implement 
SBA.  Next,  we  present  the  areas  that  programs 
can  “push”  while  implementating  SBA  activi¬ 
ties  within  their  program.  Finally,  we  discuss 
the  “pull”  programs  will  receive  from  the 
warfighting  and  programming  communities 
once  they  and  other  programs  begin 
implementing  SBA . 

Factors  Influencing  the  Use  of  SBA  - 
Domain,  Infrastructure,  and  Need 

Many  programs  are  doing  smart  things  with 
M&S,  and  many  of  these  programs  say  they 
are  already  implementing  SBA.  Others 
counter  that  those  programs  are  only  begin¬ 
ning  to  scratch  the  surface  of  what  is  pos¬ 
sible  using  a  simulation-based  approach  to 
acquisition.  They  are  just  gaining  benefits 
from  having  a  common  description  of  the 
product,  so  they  cannot  claim  to  be  imple¬ 
menting  SBA.  SBA  is  a  new  approach  that 
has  a  sliding  scale — programs  can  do  a  little, 
or  they  can  do  a  lot.  As  someone  said  at  a 
recent  conference,  SBA  is  a  journey,  not  a 


destination.  How  much  SBA  can  be  done  in  a 
program  largely  depends  on  three  factors: 

1.  the  domain  the  program  is  in; 

2.  how  well  developed  the  infrastructure  is; 
and 

3.  how  compelling  a  need  there  is  to  change 
the  current  way  of  doing  business. 

Regarding  the  first  factor,  how  much  SBA  can 
be  done  will  partly  depend  on  the  maturity  of 
the  program's  domain.  The  DoD  M&S  Glos¬ 
sary  defines  a  domain  as  “The  physical  or  ab¬ 
stract  space  in  which  entities  and  processes 
operate.  A  domain  can  be  land,  sea,  air,  space, 
undersea,  a  combination  of  any  of  the  above, 
or  an  abstract  domain,  such  as  an  n-dimen- 
sional  mathematics  space,  or  economic  or  psy¬ 
chological  domains.”^  If  the  domain  is  fairly 
mature  with  well-developed  models,  then  a 
program  has  a  much  better  chance  of  leverag¬ 
ing  others’  investments  to  their  benefit.  On  the 
other  hand,  a  program’s  knowledgeable 
industry  partners  may  still  be  using  blueprints 
and  fax  machines.  These  programs  will  prob¬ 
ably  have  to  set  their  sights  a  little  lower  in 
how  far  they  will  be  able  to  reap  the  benefits 
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of  moving  to  an  acquisition  strategy  based 
upon  the  heavy  use  of  M&S. 

The  automotive  and  aerospace  industries  are 
two  of  the  most  advanced  domains  because  of 
their  lengthy  involvement  with  M&S,  and  have 
had  good  success  in  developing  their  enter¬ 
prises  by  integrating  with  their  suppliers.  (An 
enterprise  is  defined  as  “an  arbitrarily-defined 
functional  and  administrative  entity  that  exists 
to  perform  a  specific,  integrated  set  of  missions 
and  achieve  associated  goals  and  objectives, 
encompassing  all  of  the  primary  functions 
necessary  to  perform  those  missions.”^)  Con¬ 
versely,  the  U.S.  shipbuilding  industry  has  many 
domestic  suppliers  that  do  not  have  any  exper¬ 
tise  in  the  synthetic  environment,  and  are 
unable,  therefore,  to  provide  CAD  drawings 
(much  less  digital  models)  of  their  products. 
This  impedes  the  benefits  the  prime  contrac¬ 
tors  in  the  shipbuilding  industry  can  accrue  by 
using  M&S,  since  they  have  the  additional 
expense  of  reverse  engineering  the  models  of 
their  suppliers'  products,  with  all  of  the  attendant 
worries  about  the  validity  of  the  models. 

It's  important  to  note  that  in  spite  of  this  limi¬ 
tation,  the  shipbuilding  industry  has  gained 
significant  benefits  from  using  M&S.  For 
example.  General  Dynamics  Electric  Boat 
Corporation  was  able  to  reduce  the  number 
of  pipe  hangars  on  the  New  Attack  Subma¬ 
rine  (NSSN)  from  40,000  to  about  18,000. 
Since  hangars  are  a  major  cost  item  on  a  sub¬ 
marine,  this  equates  to  a  major  cost  savings 
for  the  program.  General  Dynamics  more  than 
recouped  their  implementation  costs  because 
the  cost  reduction  on  the  first  boat  alone 
exceeded  the  entire  cost  of  modeling  it,  and 
they  are  building  three  more.^ 

The  second  factor  influencing  the  transition 
to  SBA  will  be  how  much  of  the  infrastructure 


is  in  place  to  facilitate  a  program's  entry  into 
a  simulation-based  acquisition  process.  The 
Defense  Modeling  and  Simulation  Office 
(DMSO)  is  building  many  parts  of  this  infra¬ 
structure.  For  example,  DMSO  has  created  the 
Modeling  and  Simulation  Resource  Reposi¬ 
tory  (MSRR),  which  is  a  collection  of  models, 
simulations,  object  models,  Conceptual 
Models  of  the  Mission  Space  (CMMS),  algo¬ 
rithms,  instance  databases,  data  sets,  data  stan¬ 
dardization  and  administration  products, 
documents,  tools,  and  utilities.  The  MSRR  is 
a  collection  of  resources  hosted  on  a  distrib¬ 
uted  system  of  resource  servers.  These  servers 
are  interconnected  through  the  worldwide  web 
using  the  internet  for  the  unclassified  MSRR 
and  through  the  Secret  Internet  Protocol 
Routing  Network  (SIPRNET)  for  the  classi¬ 
fied  MSRR.  The  MSRR  provides  a  layer  of 
services  that  includes  the  registration  of 
resources  and  users,  description  and  quality 
information  of  resources,  and  specialized 
search  capabilities."^  DMSO  has  also  created 
the  Modeling  and  Simulation  Operational 
Support  Activity  (MSOSA),  which  assists  DoD 
activities  in  meeting  their  M&S  needs  by  pro¬ 
viding  operational  advice  and  facilitating 
access  to  M&S  information  and  assets.  The 
MSOSA  is  a  contractor-staffed  activity  oper¬ 
ating  under  the  direction  of  the  DMSO  Di¬ 
rector  of  Operations.^  Programs  needing  M&S 
help  or  assistance  should  contact  the  MSOSA. 
Contact  information  is  available  through  the 
DMSO  website  (mentioned  previously  in  the 
preface). 

DMSO  is  also  leading  a  DoD-wide  effort  to 
establish  a  common  technical  framework  to 
facilitate  the  interoperability  of  all  types  of 
models  and  simulations,  among  themselves 
and  with  command,  control,  communica¬ 
tions,  computers  and  intelligence  (C4I)  sys¬ 
tems,  as  well  as  to  facilitate  the  reuse  of 
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M&S  components.  This  Common  Technical 
Framework  (CTF)  consists  of  three  pieces:  the 
CMMS,  Data  Standards  (DS),  and  the  High 
Level  Architecture  (HLA).^ 

The  mission  of  the  CMMS  is  to  develop  a 
conceptual  model  of  the  mission  space  for 
each  DoD  mission  area  to  provide  a  common 
basis  for  development  of  consistent  and 
authoritative  M&S  representations.  Its  pur¬ 
pose  is  to  provide  designers  with  an  evolvable 
and  accessible  framework  of  tools  and 
resources  for  conceptual  analysis.  The  mission 
space  structure,  tools,  and  resources  will  pro¬ 
vide  both  an  overarching  framework  and 
access  to  the  necessary  data  and  detail  to  per¬ 
mit  development  of  consistent,  interoperable, 
and  authoritative  representations  of  the  envi¬ 
ronment,  systems,  and  human  behavior  in 
DoD  simulation.  Using  this  framework, 
designers  will  be  able  to  develop  a  clear  picture 
of  what  they  wish  to  represent  in  order  to  pro¬ 
duce  a  workable  model  or  simulation  for  any 
application.  This  picture  will  be  multi-dimen¬ 
sional  and  must  include  a  depiction  of  the 
entities,  actions,  and  interactions  that  must  be 
represented.  There  will  be  several  CMMS  cor¬ 
responding  to  broad  mission  areas  (such  as 
conventional  combat  operations,  other  mili¬ 
tary  operations,  training,  acquisition,  and 
analysis).’ 

The  second  piece  of  the  CTF  is  the  M&S  Data 
Standards  Program.  The  mission  of  the  M&S 
DS  Program  is  to  enable  data  suppliers  to  pro¬ 
vide  the  M&S  community  with  cost-effective, 
timely,  and  certified  data  to  promote  reuse  and 
sharing  of  data;  interoperability  of  models  and 
simulations  within  themselves  and  with  the 
warfighter’s  C4I  systems;  and  improved  cred¬ 
ibility  of  M&S  results.  The  strategic  objective 
is  to  establish,  promulgate,  and  oversee 
policies,  procedures  and  methodologies  for 


M&S  data  requirements;  data  standards;  data 
verification,  validation,  and  certification; 
authoritative  data  sources  (ADS)  and  data 
security  to  provide  quality  data  as  common 
representations  of  the  natural  environment, 
systems,  and  human  behavior.  The  Data 
Program  is  organized  into  four  areas:  Data 
Engineering  (including  Data  Interface 
Format  (DIF)  and  tool  development), 
Authoritative  Data  Sources,  Data  Quality, 
and  Data  Security.^ 

Within  the  Data  Engineering  area,  the  Syn¬ 
thetic  Environment  Data  Representation  and 
Interchange  Specification  (SEDRIS)  effort  is 
significant  for  facilitating  SBA.  Currently, 
there  is  no  uniform  and  effective  standard 
mechanism  for  interchanging  synthetic 
environments  between  M&S  applications. 
SEDRIS  will  support  the  unambiguous  inter¬ 
change  of  data  between  database  generation 
systems  by  using  a  standard  application 
programmer’s  interface  that  will  allow  the  data 
to  be  captured  from  the  producer’s  native  for¬ 
mat  with  accompanying  explanatory  metadata. 
Although  SEDRIS  by  itself  will  not  solve  all 
interoperability  problems,  it  will  provide  the 
technology  to  enable  solutions.^ 

The  third  piece  of  the  CTF  is  the  High  Level 
Architecture  (HLA)  program.  The  HLA  is  a 
com-posable  approach  to  constructing  simu¬ 
lations  that  recognizes  that  no  single,  mono¬ 
lithic  simulation  can  satisfy  the  needs  of  all 
users;  all  uses  of  simulations  and  useful  ways 
of  combining  them  cannot  be  anticipated  in 
advance;  and  future  technological  capabilities 
and  a  variety  of  operating  configurations  must 
be  accommodated.^®  The  HLA  provides  a 
common  framework  within  which  specific  sys¬ 
tem  architectures,  or  federations,  are  defined 
by  three  components.  These  are: 
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1.  Ten  rules  that  define  the  relationships 
among  the  federation  components; 

2.  An  object  model  template  that  specifies 
the  form  in  which  simulation  elements  are 
described; 

3.  An  interface  specification  that  describes 
the  way  simulations  interact  during 
operation.^^ 

A  federate  is  a  member  of  a  HLA  federation, 
which  may  include  federation  managers;  data 
collectors;  real  world  (“live”)  systems  such  as 
instrumented  ranges,  sensors,  or  C4I  systems; 
simulations;  passive  viewers;  and  other 
utilities.^^ 

The  MSRR,  MSOSA,  and  CTF  are  all  pieces 
of  the  SBA  infrastructure  that  facilitate  the 
entry  of  programs  into  SBA.  (We^ll  have  more 
to  say  about  standards  and  interoperability  in 
Chapter  7,  Challenges  to  Implementing  SBA.) 
At  this  stage,  many  of  the  tools  needed  to  do 
enterprise-wide  simulation  are  at  an  imma¬ 
ture,  “micro”  level,  and  are  not  ready  to  handle 
“macro”  jobs. 

The  big  three  auto  makers  have  been  solving 
their  enterprise  data  interoperability  needs  by 
dictating  that  their  first-tier  suppliers  must  use 
their  computer-aided  design  packages  for  de¬ 
velopment  and  submission  of  their  products. 
(Chrysler  uses  Dassault  Systemes’  CATIA, 
Ford  uses  SDRC’s  Ideas,  and  General  Motors 
uses  Unigraphics  Solutions,  Inc.’s  package). 
The  commercial  side  of  The  Boeing  Company 
has  also  standardized  with  CATIA.  Others  are 
choosing  to  solve  their  enterprise  needs  by 
using  standards  for  the  universal  exchange  of 
product  information.  For  example,  the  Stan¬ 
dard  for  the  Exchange  of  Product  Data 
(STEP)  is  an  emerging  international  standard 


for  representing  data  about  products  that  pro¬ 
vides  a  set  of  standard  definitions  for  the  data 
throughout  the  product  life  cycle.^^  Using 
STEP,  The  Boeing  Company  (military  side)  is 
able  to  successfully  exchange  design  informa¬ 
tion  between  its  St.  Louis  facility  (which  uses 
Unigraphics  to  design  the  Joint  Strike 
Fighter’s  fore  body)  with  its  Seattle  facility 
(which  uses  CATIA  in  its  role  as  systems  inte¬ 
grator).^'*  General  Dynamics  Land  Systems 
(GDLS)  uses  both  Pro-E  and  Computervision 
for  their  design  software,  while  their  vendors 
mostly  use  Autocad,  Unigraphics,  and  Pro-E. 
GDLS  maintains  interoperablity  by  focusing 
their  efforts  on  ensuring  the  accuracy  of  the 
data.*^  Likewise,  United  Defense  Limited 
Partnership  is  using  various  commercially 
available  tools  (e.g.  Pro-E  for  solid  modeling 
and  Corypheous  for  visualization)  and  link¬ 
ing  them  together  to  get  the  job  done.*^  The 
effort  a  program  must  devote  to  ensuring 
seamless  interoperability  between  the  prod¬ 
ucts  of  M&S  tools  will  depend  on  factors  such 
as  the  number  of  different  M&S  tools  used, 
their  degree  of  compatibility  and  the  level 
which  they  interface.  Translation  efforts  in  the 
examples  cited  above,  range  from  being  a  nui¬ 
sance  to  being  major  problems  that  require 
full  time  support. 

Regardless  of  the  method  used,  programs 
need  a  way  to  communicate  freely  in  their  in¬ 
tegrated  data  environments.  HLA  compliance 
for  simulations  has  been  mandated  by  OSD 
for  30  September  2000  and  non-compliant 
development  and  modification  of  simulations 
must  cease  by  30  September  1998;  however, 
“HLA  is  not  an  interoperability  ‘magic  wand.’ 
That  is,  HLA  will  not  automatically  make 
every  simulation  suitable  for  federating  with 
every  other  simulation,  nor  guarantee  a  valid, 
meaningful  exchange  of  information  across  the 
federation  . . .  but  the  HLA  does  provide  the 


critical  technical  foundation  for  the  interop¬ 
erability  of  simulations  among  themselves,  and 
with  live  systems.”^^  Said  another  way,  HLA 
will  provide  the  common  “plug”  within  a 
federation,  but  not  guarantee  that  models 
from  different  federations  will  “play”  to¬ 
gether  in  simulation.  When  examining  the 
possibilities,  the  real  answer  may  often  be  that 
applications  need  to  be  “compatible  but  not 
necessarily  common.”^^ 

A  final  factor  influencing  the  transition  to  SBA 
will  be  how  compelling  a  need  there  is  to 
change  the  current  way  of  doing  business.  Cri¬ 
sis  is  often  the  reason  for  change  and  has  been 
a  force  driving  many  programs  to  turn  to 
increasing  and  more  innovative  uses  of  M&S. 
Certainly  this  is  the  case  for  the  Army’s 
Comanche  helicopter  program.  The  program 
office  used  M&S  extensively  to  conduct  a  com¬ 
petitive  fly  off  in  which  two  competing  teams 
used  man-in-the-loop  simulation  instead  of 
building  costly  protot5rpe  aircraft.  In  addition, 
M&S  activities  allowed  program  management 
to  work  within  budget  constraints  and  main¬ 
tain  a  viable  program  throughout  developmen¬ 
tal  test  and  evaluation,  with  only  one  flying 
prototype.^^  In  the  automotive  industry  this 
was  most  apparent  with  Chrysler,  which  was 
on  the  verge  of  bankruptcy  in  the  early  90s  with 
many  employees  working  half  days.  In 
response  to  this  crisis,  Chrysler  built  a  new 
technology  and  design  center  and  made  a  sig¬ 
nificant  switch-over  to  using  computer-aided 
engineering.  A  workforce  that  saw  and  sup¬ 
ported  the  need  for  change  made  this  possible. 
Similarly,  within  the  DoD  there  is  a  compel¬ 
ling  need  to  find  a  better  and  more  affordable 
way  of  acquiring  systems.  It  is  unwise  to  con¬ 
tinue  business  as  usual  in  light  of  the  rapidly 
evolving  M&S  technology  that  will  support 
SBA  as  a  better  alternative. 


SBA  Push  and  SBA  Pull 

The  domain  of  the  program,  the  maturity  of 
the  infrastructure,  and  the  degree  of  need  will 
all  influence  how  much  SBA  can  be  imple¬ 
mented  in  a  program.  But  the  key  point  is  that 
any  program  can  start  moving  down  the  path 
to  implementing  SBA.  This  “SBA  Push”  is 
what  programs  are  doing  within  their  enter¬ 
prise  to  implement  SBA,  given  the  existing 
state  of  the  domain,  infrastructure,  and  need. 

Likewise,  the  “SBA  Pull”  is  the  expectation 
and  demand  for  SBA  from  outside  agencies 
(for  example,  the  warfighting  and  resource 
allocation  communities).  Dollar  for  dollar, 
programs  that  are  implementing  SBA  will 
be  perceived  as  better  programs,  because  of 
the  increased  visibility,  superior  insight,  and 
subsequent  “better,  faster,  and  cheaper” 
products. 

SBA  Push 

We’ll  start  first  by  looking  at  how  programs 
can  expand  the  envelope  by  moving  down  the 
path  to  implementing  SBA.  Regardless  of 
where  a  program  is  in  its  life  cycle,  it  can  ben¬ 
efit  by  initiating  SBA  practices.  It  is  usually 
not  cost-effective  for  a  program  to  wait  until 
everything  is  completely  available  and  in  place 
before  starting  to  use  SBA.  It’s  more  impor¬ 
tant  to  get  started,  achieve  some  successes, 
learn,  and  continue  to  build.  A  good  analogy 
is  that  of  building  a  housing  subdivision.  Once 
the  overall  plan  is  in  place,  there’s  a  tradeoff 
between  the  cost-efficiencies  of  scale  and  vol¬ 
ume  and  the  investment  required.  Usually  this 
means  building  a  few  houses,  selling  them  to 
generate  cash  flow  to  finance  additional 
houses  and  infrastructure,  and  continuing  to 
expand  throughout  the  subdivision.  An  added 
benefit  to  this  approach  is  the  learning  that 
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occurs  as  each  house  is  built.  The  last  house 
should  be  significantly  better,  take  less  time, 
and  cost  less  to  build  than  the  first. 

Defense  acquisition  programs  are  normally 
inventing  or  reinventing  a  new  product  and 
inventions  do  not  have  reliability — reliability 
is  achieved  through  history,  use,  and  time.  It’s 
important,  therefore,  to  get  started  in  SBA  in 
order  to  continually  and  incrementally 
improve  the  processes,  rather  than  waiting 
until  everything  is  in  place.  The  movement 
towards  SBA  will  be  a  progression.  It  has  been 
stated  that  “the  concept  is  revolutionary,  but  the 
implementation  will  be  evolutionary.”^^ 

An  important  consideration  for  the  program 
manager  is  to  not  oversell  SBA.  As  noted  by 
one  industry  manager,  the  rate  of  change 
towards  SBA  is  a  concern.  The  community 
must  accurately  portray  the  direction,  the 
speed  of  advance,  and  the  capabilities  of  the 
final  end  state  of  SBA  so  that  expectations  are 
in  line  with  reality.  There  will  be  incremental 
changes  and  payoffs.  Creating  unrealistic 
expectations  can  negate  any  potential  benefits. 
In  particular,  there  may  be  many  things  that 
cannot  be  modeled  well  enough  to  get  the 
information  needed. 

Does  it  require  an  up-front  investment  for  pro¬ 
grams  to  adopt  SBA?  Certainly  there  are  ini¬ 
tial  investments  required  in  terms  of  the  com¬ 
puter  and  software  purchases,  as  well  as 
workforce  training.^^  There  are  other  costs,  in¬ 
cluding  application  software  changes  requiring 
the  updating  and  revalidation  of  simulations. 

Much  has  been  said  about  The  Boeing 
Company’s  failure  to  recoup  its  approximately 
$1.5B  to  $2B  investment  for  the  full  CAD/ 
CAM  system  used  on  the  111  airplane.  How¬ 
ever,  their  automated,  simulation-based 


design  and  build  process  enabled  them  to  de¬ 
liver  a  completely  reliable,  operational  aircraft 
on  day  one,  a  first-of-its-kind  in  the  commer¬ 
cial  aircraft  industry.  In  the  past  it  usually  took 
a  year  after  delivery  to  work  out  all  the  soft¬ 
ware  bugs,  maintenance  issues,  and  parts  dis¬ 
tribution  for  a  new  aircraft.  Boeing’s  instant 
success  with  the  first  aircraft  has  stimulated 
more  sales  and  has  further  benefits  that  extend 
across  improved  operations  for  the  entire 
Boeing  commercial  aircraft  workforce  and 
procedures.  The  company  continues  to  remain 
well  positioned  to  outstrip  their  competition 
with  faster  aircraft  upgrades,^^  With  better 
customer  satisfaction,  lower  operating  costs, 
and  lower  maintenance  costs,  Boeing  expects 
to  sell  more  111  aircraft.  They  are  also  able  to 
use  this  technology  on  other  aircraft  for  no 
development  costs.  Finally,  if  Boeing  hadn’t 
got  the  aircraft  to  market  as  quickly.  Airbus 
might  have  taken  some  of  the  sales.  Benefits 
like  these  are  difficult  to  factor  into  the 
investment  decision  with  hard  data  because 
there  is  no  way  to  predict  the  outcome  had 
traditional  processes  been  used.  This  “cost 
avoidance”  or  opportunity  cost  issue  makes  it 
difficult  to  determine  the  return  on  investment 
for  switching  to  the  SBA  approach.  Programs 
can  show  cost  avoidance  figures  resulting  from 
the  building  of  fewer  prototypes,  or  more  ef¬ 
ficient  tests,  or  the  need  for  only  one  flying 
testbed.  But  it  is  very  difficult,  if  not  impos¬ 
sible,  to  put  dollar  values  on  risk  reduction  or 
a  greatly  improved  product. 

In  any  event,  most  programs  probably  will  not 
receive  extra  funding  to  change  over  to  this 
new  SBA  approach.  Reality  is  a  matter  of 
reallocation  of  the  existing  budget  to  obtain 
the  best  value,  and  determining  when  that 
value  will  be  received.  Again,  this  will  re¬ 
quire  a  tradeoff  analysis  to  balance  specific 
needs  with  the  available  resources.  The 


Army  dictates  the  use  of  a  Simulation  Sup¬ 
port  Plan  (SSP)  to  effectively  manage  and  in¬ 
tegrate  the  use  of  M&S  within  acquisition  pro- 
grams.^^  The  Air  Force  handles  this  tradeoff 
analysis  by  requiring  programs  to  reflect  their 
M&S  strategy  and  requirements  in  the  appro¬ 
priate  acquisition  documentation,  such  as  Op¬ 
erational  Requirements  Documents,  Test  and 
Evaluation  Master  Plans,  and  Single  Acquisi¬ 
tion  Management  Plans.^'*  The  objective  of 
these  requirements  is  the  same:  to  determine 
the  high-leverage  areas  that  simulation  can 
enhance,  while  at  the  same  time  planning  to 
build  from  these  successes. 

High-leverage  areas  can  vary  greatly  from  one 
program  to  another.  Factors  that  can  influence 
a  program's  high-leverage  areas  are  such 
things  as  the  program's  developmental  time 
line  and  the  program's  strategies  and  goals. 
For  example,  rather  than  baselining  a  system 
first  and  then  creating  a  trainer,  programs  are 
creating  early-on  test-beds  that  provide  the 
opportunity  to  iterate  and  immerse  the  user, 
which  can  ultimately  grow  into  a  trainer.^ 
Through  the  use  of  the  virtual  environment 
the  Crusader  program,  with  a  simulation  bud¬ 
get  of  $9M,  was  able  to  reduce  the  program's 
requirement  for  physical  prototypes  by  six.  At 
a  price  tag  of  $50M  each,  the  program  real¬ 
ized  a  total  cost  avoidance  of  $300M.^  Strate¬ 
gies  and  goals  also  help  identify  programmatic 
high-leverage  areas.  Chrysler  has  reduced 
their  cost  of  doing  business  by  $800M  by  tak¬ 
ing  eight  months  out  of  the  cycle  time  through 
the  use  of  the  virtual  environment  during  de¬ 
sign  development.  Their  prototypes  are  now 
as  good  as  the  first  production  cars  were  when 
they  were  using  the  traditional  production 
method.^’  General  Motors  does  80  percent  of 
its  manufacturing  in-house  (compared  with 
Chrysler's  20  percent  and  Ford's  50  per¬ 
cent).  GM's  costs,  therefore,  are  associated 


with  tooling,  and  they  must  concentrating  on 
increasing  their  confidence  level  before  com¬ 
mitting  to  tooling.^  This  goal  of  increasing  the 
confidence  level  in  the  design  is  supported 
through  the  use  of  SBA  practices.  Ford,  on 
the  other  hand,  concentrates  on  a  global,  dis¬ 
tributed  engineering  capability,  which  empha¬ 
sizes  the  importance  of  product  information 
management.2^  Designing  in  the  virtual  envi¬ 
ronment  supports  this  strategy  of  distributed 
engineering  and  information  management. 

Many  of  today's  programs  focus  primarily  on 
the  design  development  portion  and  not  the 
full  life  cycle  of  a  system  and  indeed,  this  is  an 
important  first  step  in  achieving  the  full  ben¬ 
efits  of  SBA.  Within  systems,  we're  just 
beginning  to  simulate  beyond  the  traditional 
performance  issues  to  address  the  entire  life 
cycle's  cost  issues  during  the  design.  These 
include  manufacturability,  supportability, 
lethality,  sustainability,  mobility,  survivability, 
flexibility,  interoperability,  reliability,  and 
affordability.  One  of  the  nearest  term  capa¬ 
bilities  that  will  benefit  many  programs  is  “vir¬ 
tual  manufacturing" — the  ability  to  look  at  the 
manufacturing  issues  during  design. 

The  F-22  program  is  looking  at  supportability 
issues  by  using  a  3-D  model  that  has  enabled 
two  mechanics  to  influence  the  design  by  mak¬ 
ing  critical  inputs  long  before  any  hands-on 
prototypes  were  available.^  The  JSF  program 
is  developing  the  Joint  Interim  Mission  Model 
(JIMM)  to  merge  two  legacy  models,  enabling 
test  and  training  inputs  to  their  early  model¬ 
ing  efforts  until  JMASS  is  available.^^  The  Cru¬ 
sader  program,  together  with  the  combat 
development  community,  has  jointly  con¬ 
ducted  a  series  of  simulated  warfighting 
experiments  called  Concept  Evaluation  Pro¬ 
grams  (CEPs),  which  provided  early  warfighter 
insights  into  the  changing  tactics,  techniques. 
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and  procedures  associated  with  the  new 
weapon  system.^^  During  the  redesign  of  the 
Boeing  737  aircraft,  Boeing  found  that 
digitizing  older  systems  where  the  design  was 
only  on  blueprints  can  still  pay  big  dividends. 
They  did  a  full  Digital  Product  Assembly  and 
Digital  Product  Definition  for  only  those  parts 
that  were  new  (the  wing,  empennage,  and 
main  landing  gear).  Although  the  digitization 
of  the  old  parts  was  tedious,  Boeing  found  it 
to  be  worthwhile.^^  The  JSF  created  cost  curves 
for  each  alternative  during  its  Analysis  of 
Alternatives,  which  allowed  operational 
tradeoffs  using  CAJV.^ 

Using  the  CAIV  concept,  it  is  critical  for 
a  program  manager  to  know  how  design 
decisions  will  impact  on  overall  program  costs. 
Some  programs  have  developed  cost  models 
based  on  historical  costs  of  similar  type 
systems.  A  good  example  is  the  CVN-77  air¬ 
craft  carrier  program  that  looked  at  previous 
ships  to  determine  what  the  cost  drivers  were 
throughout  the  life  cycle  of  a  ship,^^  This  iden¬ 
tified  where  the  CVN-77  program  should 
target  attention  to  achieve  the  most  cost 
reduction. 

The  Crusader  program  has  pretty  high  confi¬ 
dence  in  its  cost  models  and  has  established 
an  Ownership  Cost  Working  Group  consist¬ 
ing  of  representatives  from  the  contractor, 
user  community,  and  the  program  office  to 
identify  life  cycle  cost  drivers  and  methods  to 
eliminate  or  minimize  them.  During  the  early 
phase  of  the  Crusader’s  development,  an  in¬ 
dependent  study  group  was  commissioned  to 
analyze  the  program’s  requirements  and  de¬ 
sign  in  order  to  provide  recommendations 
balancing  weight,  cost,  and  performance.  To 
facilitate  this  process,  a  model  was  developed. 
The  model  was  based  on  interviews  with  de¬ 
sign  engineers  and  subject  matter  experts.  The 


model  showed  the  overall  change  in  force  ef¬ 
fectiveness  based  on  incremental  increases  or 
decreases  in  weight,  cost  ,and  performance.^^ 
The  program  manager  and  user  representa¬ 
tive  are  able  to  use.  this  information 
collaboratively  in  making  tradeoff  decisions. 

An  SBA  approach  enables  early  identification 
of  the  key  drivers  (such  as  cost,  schedule,  and 
performance)  throughout  the  system’s  life 
cycle,  by  simulating  systems  and  environments 
external  to  the  system.^^  The  Longbow  Hellfire 
missile  program  eliminated  its  fly-to-buy  cri¬ 
teria  by  the  close  coupling  with  their  contrac¬ 
tor  enabled  by  using  the  Simulation/Test  Ac¬ 
ceptance  Facility  (STAF)  to  finalize  the  accep¬ 
tance  test  procedures  long  before  the  produc¬ 
tion  phase.^^ 

The  High  Mobility  Artillery  Rocket  System 
(HIMARS)  program  is  an  example  of  how 
valuable  M&S  development  and  reuse  is 
becoming  to  program  managers.  The 
HIMARS  is  a  multiple  launch  rocket  system 
that  uses  a  five-ton  truck  as  its  mobility  plat¬ 
form.  The  HIMARS  program  manager 
decided  that  using  M&S  was  the  best  approach 
for  development  of  the  HIMARS  rocket 
launcher.  No  model  of  a  five-ton  truck  existed, 
but  a  model  was  needed  for  integration  efforts 
if  M&S  was  to  be  used  to  develop  the  rocket 
launcher.  The  HIMARS  program  spent  $500K 
of  a  $300M  budget  to  build  a  5-ton  truck  model 
from  the  Family  of  Tactical  Vehicles  (FMTV) 
program,  and  is  willingly  providing  this  model 
to  other  programs  that  ask  for  it.  Other  pro¬ 
grams  in  the  Tank  and  Automotive  Command 
can  now  build  off  of  this  initial  model  to 
develop  models  of  the  other  24  variants  of  2.5- 
ton  and  five-ton  vehicles  that  have  85  percent 
commonality^^  at  much  less  expense  than  if 
each  program  were  to  create  a  new  model. 
Later  on,  the  Military  Traffic  Management 
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Command  (MTMC)  used  the  five-ton  truck 
model  to  conduct  rail  impact  analysis  during 
transportability  testing.  MTMC  was  also  able 
to  use  the  model  to  conduct  air  and  ship 
impact  analysis."^  Users  outside  the  Tank  and 
Automotive  community  will  also  be  able  to 
benefit  when  this  model  is  added  to  DMSO’s 
MSRR,  and  is  searchable  and  readily  avail¬ 
able.  What  started  out  as  a  requirement  to 
model  a  single  vehicle  for  one  program  has 
grown  into  the  potential  for  influencing  many 
programs  and  aiding  the  requirements  of  other 
organizations. 

Many  companies  and  programs  are  using  a 
visualization  room  to  bring  team  members 
together  for  the  free  exchange  of  ideas 
between  disciplines.  Although  much  of  the 
work  in  an  integrated  data  environment  can 
be  distributed  and  decentralized,  a  visualiza¬ 
tion  room  visually  presents  the  digital  data  in 
forms  that  a  team  can  use  as  a  group.  Con¬ 
cepts  and  work-in-progress  can  be  displayed, 
analyzed,  and  debated  by  integrated  product 
teams,  even  if  they  are  geographically  dis¬ 
persed.  Work  can  be  imported  electronically 
to  conduct  design  reviews  or  to  immerse  a 
customer  for  feedback.  The  Marine  Corps’ 
Advanced  Amphibious  Assault  Vehicle 
(AAAV)  program  has  an  excellent  visualiza¬ 
tion  room.  It  was  built  as  an  integral  part  of 
the  entire  AAAV  facility,  which  includes  gov¬ 
ernment  and  contractor  personnel.  The 
visualization  room  provides  the  IPPD  team  a 
location  to  discuss  issues  while  viewing  3-D 
representations  of  the  vehicle  or  any  compo¬ 
nent.  If  the  team  needs  further  clarification 
of  an  issue  they  can  move  to  the  adjacent  high- 
bay  area  to  look  at  a  mockup  or  the  physical 
prototype.  This  capability  is  invaluable  for 
quickly  and  effectively  resolving  program 
issues. 


In  building  the  777  commercial  airliner, 
Boeing  used  an  application  called  FlyThru  that 
provides  a  three-dimensional  view  of  the 
airplane  as  it  is  being  designed.  This  allows 
errors  to  be  detected  and  fixed  prior  to  com¬ 
mitting  to  expensive  manufacturing  processes. 
The  fit  of  parts  and  systems  on  the  111  air¬ 
plane  was  20  times  better  than  what  had  been 
normally  achieved  in  the  past.  According  to 
Boeing’s  manager  of  Visualization  Tools,  “Be¬ 
ing  able  to  digitally  build  the  entire  plane  and 
see  the  parts  of  the  plane  before  building  it, 
was  the  biggest  money  saver  for  the  777.”^^ 

SBAPuIl 

SBA  Pull  (as  we  described  at  the  beginning  of 
the  chapter)  is  the  expectation  and  demand  from 
others  once  they  see  the  advantages  of  SBA 

From  program  inception,  the  warfighter  com¬ 
munity  will  become  an  active  participant  in 
program  design  and  development.  Long 
before  any  physical  prototypes  are  available, 
virtual  prototypes  of  system  concepts  will  be 
able  to  participate  in  exercises  that  demon¬ 
strate  the  battlefield  effects.  Logisticians  will 
be  able  to  quantify  the  cost  effectiveness  of 
making  the  system  more  supportable  and  work 
with  the  design  team  to  find  the  most  cost- 
effective  mix  of  performance  and  support 
parameters.  Using  visualization  tools  such  as 
three-dimensional  solid  modeling,  people  with 
real  field  experience  can  also  provide  real¬ 
time  user  feedback  on  design  iterations, 
because  they  will  be  able  to  “see”  and  interact 
with  the  design  as  it  matures. 

This  user  feedback  will  begin  to  extend  beyond 
the  traditional  service  boundaries  of  the  spon¬ 
soring  organization.  Programs  will  begin  to 
look  at  how  their  system  will  interact  with 
other  systems  on  the  battlefield  by  interfacing 
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those  system  models  on  a  virtual  battlefield. 
The  user  will  be  able  to  begin  using  the  new 
system  in  the  expected  environment  and 
assessing  any  potential  incompatibilities  or 
unforeseen  circumstances  that  could  be 
averted  in  the  design.  Programs  will  be  able 
to  use  each  others’  models  to  get  the  design 
information  they  need.  The  combatant  com¬ 
mands  can  also  be  immersed  at  various  inter¬ 
vals  as  the  design  matures  and  provide  their 
assessment  and  feedback  on  how  well  the  sys¬ 
tem  is  meeting  their  expectations.  There  will 
be  better  dialogue  with  the  resource  alloca¬ 
tion  community  since  the  program  will  have 
much  improved  tools  to  keep  the  cost  of  the 
system  affordable.  In  exploring  the  cost  driv¬ 
ers  over  the  entire  system’s  life,  they  will  also 
have  much  more  accurate  projections  of  the 
system’s  cost.  Regardless  of  who  is  interfac¬ 
ing  with  the  program,  these  outside  agencies 
will  have  a  much  enhanced  ability  to  see  what 
the  program  is.  Indeed,  many  programs  noted 
that  a  significant  side  benefit  was  the  great 
marketing  tool  this  approach  provides  for  visi¬ 
tors  and  oversight  personnel,  because  program 
outsiders  could  instantly  grasp  the  import  of 
what  they  saw. 

The  Crusader  field  artillery  program  started 
a  force  effectiveness  analysis  even  before  the 
request  for  proposal  was  issued.  The  contrac¬ 
tor  conducted  trade  studies  to  optimize  the  sys¬ 
tem  within  the  overall  system,  using  multiple 
scenarios.  They  looked  at  the  problems 
encountered  (e.g.  thermal  analysis,  ammuni¬ 
tion  capacity,  reliability,  maintainability,  avail¬ 
ability,  etc.)  in  terms  of  cost  per  force  effec¬ 
tiveness.  They  discovered  that  the  Crusader 
could  do  things  the  current  system  (with  up¬ 
grades)  cannot.  For  example,  the  current  Pala¬ 
din  system  can  not  keep  up  with  the  Bradley 
armored  personnel  carrier  or  the  Ml  tank.  In 
addition,  the  Crusader  frees  up  the  Multiple 


Launch  Rocket  System  to  hit  deeper  targets. 
In  short,  the  Crusader  increased  the  opera¬ 
tional  tempo  of  the  battlefield.  Some  of  these 
insights  were  intuitive,  and  some  were  not.  The 
value  of  much  of  this  information  was  diffi¬ 
cult  to  explain,  and  the  contractor  was  caught 
between  satisfying  the  requirements  stated  in 
the  request  for  proposal  and  making  tradeoffs 
whose  value  to  the  user  he  could  only  guess 
at.  For  example,  if  he  could  decrease  the  size 
of  the  crew  from  five  people  down  to  three, 
what  was  the  change  in  total  ownership  cost? 
Were  the  cost  savings  significant  enough  to 
justify  spending  money  now  to  decrease  the 
crew  size?  What  were  the  operational  tempo 
impacts  on  the  Bradley  and  would  it  be  able 
to  get  the  necessary  information  quickly 
enough  to  the  Crusader,  which  is  40  kilome¬ 
ters  away?  These  are  the  types  of  issues  that 
the  requirement  community  needs  to  deal  with 
early.  The  earlier  in  the  design  phase  we  find 
the  answers  to  these  questions,  the  better — in 
the  past  we’ve  often  not  discovered  these 
issues  until  operational  testing,  or  even  worse 
not  until  they’re  deployed  to  the  first  unit.  By 
then  we  have  already  produced  and  fielded  the 
system,  when  if  the  problem  had  been  discov¬ 
ered  earlier  the  cost  to  fix  it  would  have  been 
considerably  less. 

While  SBA  provides  the  capability  to  integrate 
back  into  the  requirements  generation  pro¬ 
cess,  it  can  also  reach  forward  into  the  resource 
allocation  process  (the  Planning,  Program¬ 
ming,  and  Budgeting  System),  by  providing 
total  ownership  cost  information  for  the  out 
year  budgets.  For  example,  there  is  an  obvi¬ 
ous  extra  cost  associated  with  putting  simula¬ 
tors  on  board  an  aircraft  carrier,  but  it  may  be 
worth  that  cost  to  counteract  deployed  train¬ 
ing  atrophy.^  SBA  provides  a  way  to  explore 
those  costs. 


Even  low  cost  demonstrations  are  finding  it 
cost-effective  to  use  a  simulation-based 
approach.  The  Predator  unmanned  aerial 
vehicle  advanced  concept  technology 
demonstration  creatively  used  M&S  to  predict 
operational  effectiveness  and  assess  alterna¬ 
tive  force  structure  options,  and  thereby 
determined  an  optimum  system  configuration."^^ 


Many  programs  are  expanding  the  envelope 
on  what’s  possible  in  SBA,  and  they  are  doing 
it  cost-effectively.  And  although  the  tools  are 
still  limited,  these  programs  are  beginning  to 
address  the  bigger  implications  of  systems  of 
systems  issues,  where  in  the  past  we  did  not 
have  the  ability  to  address  them  at  all.  Larry 
Winslow,  Director  of  Technology  for  Boeing’s 
Phantom  Works,  summed  it  up  by  saying  “Now 
we’re  doing  Program  SBA.  In  the  future  we’ll 
be  doing  Systems  of  Systems  SBA.”"^^ 


5-11 


ENDNOTES 


1.  “A  Taxonomy  for  Warfare  Simulation 
(SIMTAX),”  Military  Operations  Research  Soci¬ 
ety  (MORS)  Report,  27  October  1989,  on-line 

hUp://ww.dinso.niil/docslib/mspoIicy/glossaiy/ 

accessed  24  June  1998. 

2.  DoD  5000.59-M,  DoD  Modeling  and  Simulation 
(M&S)  Glossary,  December  1997. 

3.  Jim  Boudreaux,  Electric  Boat  Corporation,  in¬ 
terview  by  authors.  24  February  1998. 

4.  MSRR  home  page,  on-line  http://www.msrr. 
dmso.mil/  accessed  on  25  June  1998. 

5.  MSOSA  home  page,  on-line  http://www.msosa. 
mil.inter.net/Default.htm  accessed  on  25  June 
1998. 

6.  Judith  Dahmann,  “High  Level  Architecture,” 
briefing  (1998),  See  http://www.dmso.mil/  for  a 
more  in  depth  discussion  of  the  CTF  and  its  three 
components:  CMMS,  Data  Standards,  and  HLA. 

7.  DMSO  CMMS  home  page,  on-line  http:// 
www.dmso.mil/projects/cmms/  accessed  on  26 
June  1998. 

8.  DMSO  Data  Standardization  home  page,  on-line 
http://www.dmso.mil/projects/ds/  accessed  on  26 
June  1998. 

9.  http://www.sedris.org/faq_trpl.htm 

10.  HLA  home  page,  on-line  http://hla.dmso.mil/ 
accessed  on  25  June  1998. 

11.  HLA  home  page,  on-line  http://hla.dmso.mil/ 
accessed  on  25  June  1998. 

12.  DoD  5000.59-M,  DoD  Modeling  and  Simulation 
(M&S)  Glossary,  December  1997. 

13.  PDES,  Inc.  home  page,  on  line  http:// 
pdesinc.scra.org/  accessed  on  5  May  1998. 


14.  Larry  Winslow,  Director  of  Technology,  Boeing 
Company  Phantom  Works,  interview  by  authors, 
29  April  1998. 

15.  Robert  Lentz,  General  Dynamics  Land  Systems, 
interview  by  authors,  24  February  1998. 

16.  Richard  Staiert,  United  Defense  Limited  Part¬ 
nership,  interview  by  authors. 

17.  High-Level  Architecture  (HLA)  Transition  Re¬ 
port,  March  1998,  p.  2,  on  line  http://www.dmso. 
mil/hla/policy/rept611.html  accessed  on  29  May 
1998. 

18.  Dr.  Stuart  Starr,  The  MITRE  Corporation, 
interview  by  authors,  4  March  1998. 

19.  Bergantz, /kV/. 

20.  Dr.  David  Chang,  General  Motors,  interview  by 
authors,  31  March  1998. 

21.  For  more  information  on  implementing  com¬ 
puter  aided  engineering  and  concurrent  engi¬ 
neering  within  a  company,  see  “Manager’s  Guide 
to  Computer  Aided  Engineering,”  John  Krouse, 
OnWord  Press,  1993. 

22.  John  Roundhill,  Vice  President,  The  Boeing 
Company,  remarks  at  INCOSE  Annual  Sympo¬ 
sium  in  Vancouver,  B.C.,  on  30  July  1998. 

23.  Army  Regulation  AR  5-11,  “Management  of 
Army  Models  and  Simulations,”  10  July  1997. 

24.  Air  Force  Acquisition  Policy  97A-004,  “Model¬ 
ing  and  Simulation  in  Support  of  the  Air  Force 
Acquisition  Process,  ”  28  November  1997. 

25.  Colonel  Lynn  Carroll,  USAF,  Warfighter  Train¬ 
ing  Research  Division  Chief,  Air  Force  Research 
Laboratory,  briefing  to  the  Joint  Strike  Fighter 
Mission  Level  Strategy  session,  30  March  1998. 


5-12 


26.  Armbruster, 

27.  Greg  Avesian;  Chrysler  Corporation;  Manager, 
Solutions  Marketing;  interview  by  authors,  26 
February  1998. 

28.  Chang, /biV/, 

29.  Richard  Riff,  Manager,  C3P  Project  Office,  Ford 
Motor  Company,  interview  by  authors,  23  April 
1998. 

30.  Larry  Winslow,  Director  of  Technology,  Boeing 
Company  Phantom  Works,  interview  by  authors, 
29  April  1998. 

31.  Major  General  Leslie  J.  Kenne,  Program  Man¬ 
ager,  Joint  Strike  Fighter  Program,  Remarks  at 
the  National  Defense  Industrial  Association  SBA 
Conference,  17  March  1998. 

32.  Sheaves,  ibid. 

33.  Michael  Mecham,  “Aerospace  Chases  The  Soft¬ 
ware  Boom  ”  Aviation  Week  Space  Technology, 
6  October  1997, 46-49. 

34.  Kcnnc,  ibid. 

35.  Dan  Selfridge,  Newport  News  Shipbuilding, 
“Lifecycle  Cost  and  Manning  Cost  Analysis” 
briefing  given  to  NAVSEA  PMS  392  Submarine 
TOC  IPT  Meeting  on  2  April  1998. 

36.  Jim  Shields,  Chief  of  Systems  Engineering,  Cru¬ 
sader  Program  Office,  interview  by  authors,  1  July 
1998. 


37.  AxmbrustQT,ibid. 

38.  Ibid. 

39.  http://www.army-technology.com/coiitractors/ 
transport/stewart_stevenson/index.html  ac¬ 
cessed  11  Aug  1998. 

40.  Brock  Birdsong,  Missile  Research,  Development 
and  Engineering  Center,  US  Army  Aviation  and 
Missile  Command,  phone  interview  with  author, 
29  July  1998. 

41.  Marilyn  Morgan,  “Boeing  Leads  Way  in  Virtual 
Mock-\ip'\  Boeing  News,  April  3,1998. 

42.  Peter  Cherry,  Vice  President,  Vector  Research, 
Inc.,  interview  by  authors,  27  April  1998. 

43.  CJCSI  3170.01,  Requirements  Generation 
System,  (Formerly  MOP  77),  13  June  1997. 

44.  Colonel  Lynn  Carroll,  U.S.  Air  Force,  Warfighter 
Training  Research  Division  Chief,  Air  Force  Re¬ 
search  Laboratory,  briefing  to  the  Joint  Strike 
Fighter  Mission  Level  Strategy  Session,  30  March 
1998. 

45.  MG  Kenneth  R.  Israel,  Director,  Defense  Air¬ 
borne  Reconnaissance  Office,  “Modeling  and 
Simulation  Employed  in  the  Predator  Unmanned 
Aerial  Vehicle  Program,”  on-line  http:// 
www.acq.osd.mil/daro/homepage/predms.htm 
accessed  11  December  1997. 

46.  Larry  Winslow,  Director  of  Technology,  Boeing 
Company  Phantom  Works,  interview  by  authors, 
29  April  1998. 


5-13 


5-14 


6 

A  FUTURE  STATE  OF  SBA 


^Tuture  commanders  must  be  able  to  visualize  and  create  the  %est 
fit*^  of  available  forces  needed  to  produce  the  immediate  effects  and 
achieve  the  desired  results, 

Joint  Vision  2010 


This  chapter  portrays  the  implications  of  an 
SBA  approach  for  future  defense  systems 
acquisition,  by  projecting  SBA  capabilities  and 
trends.  While  it  is  difficult  to  predict  exactly 
how  change  will  be  implemented  or  to 
describe  the  resultant  organizational  structure, 
it  is  possible  to  envision  what  an  SBA 
approach  holds  for  the  future.  Within  this 
acquisition  process  of  the  future,  systems  will 
be  thoroughly  modeled  and  simulated  prior 
to  “bending  metal.”  Beginning  with  initial  user 
needs,  requirement  definition,  through  design, 
testing,  manufacturing,  logistics,  training, 
operational  usage,  and  disposal,  the  synthetic 
environment  and  integrated  teams  will  play  a 
major  role.  Ideally,  the  acquisition  process 
becomes  a  continuum  without  a  definable 
beginning  or  end,  continually  assessing  pro¬ 
jected  threats  and  needs  against  the  ability  to 
meet  those  needs.  One  senior  acquisition 
official  put  it  best  when  he  said,  “If  the  mod¬ 
els  are  done  correctly,  the  simulations  will  tell 
us  when  we  need  to  start  a  new  program.”^ 


As  the  Joint  Vision  2010  quotation  above 
states,  commanders  must  be  able  to  project 
and  select  the  best  fit  of  systems  available  to 
accomplish  their  mission.  This  applies  to  not 
only  the  existing  systems  available  today,  but 
also  to  looking  ahead  at  the  deficiencies  and 
opportunities  in  accomplishing  future  mis¬ 
sions.  SBA  will  provide  this  capability  to 
warfighting  commanders  by  giving  them  a  syn¬ 
thetic  view  of  future  battlefields,  where  con¬ 
ceptual  systems  can  be  integrated  and  their 
operational  effectiveness  assessed. 

In  describing  this  future  process  a  key 
assumption  is  that  the  systems  acquisition  pro¬ 
cess  will  continue  to  be  centered  on  programs. 
The  purpose  of  the  acquisition  process  will  be, 
as  it  is  now,  to  produce  superior  systems  in 
response  to  mission  needs.  An  SBA  approach 
will  not  change  this  product  focus,  but  instead 
will  enhance  DoD’s  ability  to  acquire  the  “best 
fit”  of  new  systems  to  meet  the  warfighter’s 
needs.  Some  argue  that  in  order  to  achieve  the 
full  potential  of  SBA  it  is  necessary  to  move 
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from  a  “product-centric”  focus  of  systems 
acquisition,  to  a  “mission-centric”  view.^  The 
argument  is  that  during  the  earliest  stages  of 
turning  a  mission  need  into  a  material  solu¬ 
tion,  it  is  necessary  to  have  the  broadest  pos¬ 
sible  view  in  order  to  optimize  the  tradeoffs 
across  service  boundaries.  This  broad  view 
could  only  be  achieved  by  merging  similar  mis¬ 
sion  areas  across  the  services,  and  divesting 
the  services  of  control  of  the  funds  for  these 
mission  areas.  The  managers  of  these  mission 
areas  would  have  budget  and  decision 
authority  to  determine  where  best  to  invest 
funding  to  meet  user  requirements. 

But  just  because  the  technology  will  enable  the 
possibility  of  doing  business  this  way  does  not 
necessarily  make  it  desirable  or  a  necessary 
pre-condition  for  implementing  SBA.  In  fact 
it  would  be  a  major  barrier  to  implementing 
SBA,  as  the  services  have  no  intention  of  giv¬ 
ing  up  their  ability  to  determine  the 
requirements  for  the  material  solutions  to 
their  mission  needs.  This  cuts  to  the  very  foun¬ 
dation  of  how  the  services  interpret  their  roles 
and  missions.  There  is  nothing  inherent  about 
this  new  way  of  doing  business  that  requires 
the  services  to  give  up  their  right  to  determine 
their  future,  so  we  reject  the  argument  that 
implementing  SBA  requires  a  “purple,”  or 
multi-service,  acquisition  corps.  Instead,  our 
assumption  is  that  acquisition  will  remain  in 
the  province  of  each  service  and  be  program¬ 
centric.  There  are  good  reasons  for  maintain¬ 
ing  separate  services,  and  they  are  continuing 
to  learn  how  to  interoperate  and  function 
as  a  joint  team.  So  too  can  the  acquisition 
system. 

Another  assumption  is  that  a  primary  objective 
of  SBA  is  to  get  the  design  right  before  build¬ 
ing  a  system,  when  in  effect  the  design 
becomes  frozen.  Any  changes  required  after 


the  start  of  production  result  in  costly 
engineering  changes  and/or  system  modifica¬ 
tions.  If  the  design  is  kept  mostly  in  the  syn¬ 
thetic  environment  prior  to  production,  the 
cost  and  time  required  to  make  the  change  is 
significantly  reduced.  In  an  SBA  process,  the 
defining  point  is  when  the  program  moves  out 
of  the  synthetic  environment  and  begins 
production. 

The  new  ability  to  bring  systems  together  while 
they  are  still  in  design  will  be  key  to  ensuring 
that  systems  will  work  together  when  they  are 
fielded,  and  that  they  are  not  over-  or  under¬ 
designed.  Competing  new  systems  can  be 
evaluated  side-by-side  on  equal  terms  to  de¬ 
termine  the  best  alternative.  Duplication  of 
capability  within  and  across  Services  will  be 
more  readily  apparent,  resulting  in  significant 
cost  avoidance.  According  to  Dr.  Peter  Cherry, 
Vice  President  of  Vector  Research,  Inc.,  “The 
tools  we’re  using  are  getting  better,  but  now 
we  need  a  richer  context  in  which  to  make  de¬ 
cisions,  otherwise  we’ll  continue  to  only  make 
acquisition  decisions  on  the  margin.  We  make 
marginal  decisions  now — the  challenge  is  to 
provide  a  system  of  systems  context  so  we 
can  do  better  than  decisions  on  the  margin. 
Previously  we  only  made  incremental  changes 
program  by  program.  Today,  systems  of  sys¬ 
tems  require  the  total  integration  of  systems. 
We  need  to  be  able  to  walk  the  system’s  impact 
on  the  battlefield  all  the  way  from  the  platoon, 
to  the  company,  to  the  battalion  level.  We  have 
a  lack  of  the  full  understanding  of  system  sup- 
portability  issues.  We  still  talk  about  systems 
from  the  cockpit  perspective — ^we  need  to 
elevate  up  to  a  higher  level,  and  talk  to  Con¬ 
gress  about  battlefield  effectiveness,  not  how 
much  faster  or  how  stealthy  a  system  is.  For 
example,  we  need  to  be  able  to  show  the  value 
of  the  JSTARS  [Joint  Surveillance  and  Target 
Attack  Radar  System  aircraft]  to  the  artillery, 
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not  how  many  targets  it  can  track  and  it's  mean 
time  between  failure.”"^^ 

System  of  systems  issues  are  a  primary  concern 
of  the  Service  Chiefs  and  the  Joint  Require¬ 
ments  Oversight  Council  (JROC);  however, 
these  concerns  are  now  primarily  handled 
at  a  very  high  level  through  the  use  of  a 
Capstone  Requirements  Document  (CRD). 
A  CRD  is  used  to  identify  overarching  re¬ 
quirements  for  a  system,  or  several  pro¬ 
grams  that  form  a  system  of  systems.  It  con¬ 
tains  performance-based  requirements  to 
facilitate  development  of  individual  program 
requirements,"^^ 

Programs  implementing  SBA  will  be  able  to 
give  the  users  the  necessary  insight  to  balance 
these  needs  and  deficiencies  in  time  to  influ¬ 
ence  the  design,  rather  than  waiting  for  the 
Services  and  JROC  to  validate  ever-increasing 
point  design  solutions  during  Milestone 
reviews.  At  this  high  level  it's  too  late  to  impact 
the  design  cost-effectively,  as  all  of  the  assump¬ 
tions  and  tradeoff  decisions  have  already  been 
made  in  getting  ready  for  the  review.  What 
are  being  presented  are  the  results  of  the 
program  optimized  against  the  static  require¬ 
ments  dictated  in  the  Operational  Require¬ 
ments  Document  (ORD),  which  assumed  the 
user  fully  understood  the  impact  of  those 
requirements. 

With  this  in  mind,  we  will  start  by  looking  at 
the  beginning  of  a  program.  This  chapter 
discusses  a  future  state  of  SBA  by  looking 
at  four  major  phases  over  the  life  cycle  of  a 
program,  beginning  with: 

1.  generating  a  mission  need; 

2.  iterative  design  and  development  activi¬ 
ties; 


3.  production;  and 

4.  operations  and  sustainment. 

Needs  Generation  Phase 

SBA  will  encompass  more  than  what  we  today 
call  acquisition.  SBA  will  provide  a  common 
synthetic  environment  that  the  warfighting 
community  can  use  to  experiment  with  non¬ 
material  solutions  to  meet  deficiencies,  and  to 
help  them  determine  affordable  requirements. 
As  stated  in  the  Chairman  of  the  Joint  Chiefs 
of  Staff  Special  Instruction  for  the  Require¬ 
ments  Generation  System,  the  warfighter 
determines  the  need  for  a  material  or  a  non¬ 
material  solution  by  conducting  a  mission  area 
analysis  of  the  current  and  projected  capabili¬ 
ties  to  accomplish  assigned  missions.  The  Ser¬ 
vices'  first  try  to  satisfy  mission  needs  through 
nonmateriel  solutions,  such  as  changes  in  doc¬ 
trine  or  tactics.  The  Services  will  be  able  to 
use  updated  models  and  simulations  provided 
by  the  SBA  community  as  part  of  their  mis¬ 
sion  area  analysis.  If  a  nonmateriel  solution  is 
deemed  not  feasible,  the  need  is  translated 
into  a  Mission  Need  Statement  (MNS)  which 
is  expressed  in  broad  operational  terms.^  As 
the  MNS  is  documented,  validated  and 
approved,  the  SBA  process  will  bring  the 
warfighter  and  acquisition  communities 
together  from  the  very  beginning  of  program 
definition.  Members  of  the  acquisition  com¬ 
munity  will  have  a  better  understanding  of  the 
warfighter's  need  because  they  participated  in 
the  MNS  development.  Information  gained 
from  non-material  experiments  can  provide 
the  acquisition  community  insight  into  the 
warfighter's  needs,  and  any  models  developed 
may  be  of  use  in  searching  for  a  material 
solution. 
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The  generation  of  an  MNS  marks  the  end  of 
the  needs  generation  phase  in  the  SBA  pro¬ 
cess.  The  warfighting  and  acquisition  commu¬ 
nities  will  determine  the  members  of  the  IPT 
who  will  collaborate  to  generate  and  define 
the  operational  requirements  during  the  next 
phase  of  iterative  design  and  development. 

Iterative  Design  and 
Development  Phase 

The  second  major  phase  of  a  program  in  this 
future  state  of  SBA  is  iterative  design  and 
development.  As  discussed  in  Chapter  One, 
the  early  decisions  in  a  program  are  critical 
because  to  a  large  extent  they  will  determine 
system  effectiveness  and  what  the  system  will 
cost  throughout  its  life  cycle.  The  iterative 
process  of  determining  requirements  sets  the 
stage  for  the  TOC  of  the  program.  SBA 
enables  the  acquisition  community  to  support 
the  warfighters  in  determining  affordable 
requirements,  and  provides  a  vehicle  to  get 
industry  involved  during  early  concept  de¬ 
velopment.  Rather  than  locking  in  require¬ 
ments  before  understanding  them  fully,  the 
requirements  group  in  a  program  office 
develops  simulations  to  conduct  engineering 
trade  studies  that  quantify  capability  versus 
cost.  Flexibility  in  design  is  maintained  as  long 
as  possible  by  keeping  the  requirements  fluid 
until  the  cost  impacts  of  each  are  well  under¬ 
stood.  The  warfighters,  industry,  and  program 
office  personnel  are  an  integrated  team  that 
makes  informed  decisions  regarding  capabil¬ 
ity  versus  cost  issues  across  the  entire  life  cycle 
of  the  system. 

During  this  phase,  models  and  simulations 
representing  new  and  revolutionary  concepts 
and  systems  are  evaluated  in  a  series  of 
iterative  “what  if”  trade  off  analyses.  New  con¬ 
cepts  are  evaluated  in  a  virtual  environment 


to  determine  their  operational  impact  and 
system  effectiveness.  Cost  models  provide  the 
TOC  impacts  of  competing  concepts  and 
designs. 

New  conceptual  models  are  evaluated  to 
determine  if  they  provide  significant  opera¬ 
tional  benefit,  and  promising  concepts  are 
developed  further.  As  confidence  in  the 
models  grows,  many  other  issues  can  be 
addressed  concurrently.  Initial  cost  models  of 
a  design  provide  a  rough  order  of  magnitude 
(ROM)  estimate  of  the  TOC.  As  the  design 
matures  these  ROMs  are  refined  and  become 
more  accurate.  Using  an  integrated  visual  rep¬ 
resentation  of  the  system,  functional  experts 
on  the  IPPD  team  make  their  requirements 
and  concerns  known  to  all  IPPD  members,  and 
work  directly  with  each  other  in  determining 
the  optimum  design.  Training  requirements 
are  addressed  and  decisions  made  whether  to 
build  a  stand-alone  trainer  or  to  have  training 
embedded  in  the  system.  Logisticians  and 
testers  address  supportability  and  testability 
issues  and  compare  results  between  compet¬ 
ing  conceptual  systems.  The  transportation 
community  evaluates  transportability  of  the 
system  design.  Manufacturing  issues  are 
addressed  by  designing  a  virtual  factory.  Itera¬ 
tions  continue  until  a  high  level  of  confidence 
is  reached  in  the  design,  at  which  time  the 
system  is  submitted  for  a  production  decision. 
If  approved,  all  models,  simulations  and  data 
are  passed  on  for  use  in  producing  the  system. 

There  should  be  little  chance  of  a  system  being 
disapproved  during  a  production  decision, 
because  all  stakeholders  have  frequent  oppor¬ 
tunities  for  visibility  and  feedback  during  sys¬ 
tem  design.  Periodic  reviews  of  the  system’s 
progress  will  be  possible  by  immersing  the 
audience  using  tools  such  as  visualization 
rooms.  Problem  areas  can  be  resolved  quickly 
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by  bringing  in  all  stakeholders  to  buy  off  on 
critical  affordability  decisions  as  part  of  the 
design  tradeoffs. 

Oversight  activities  are  necessary  to  ensure 
programs  are  producing  the  best  possible 
weapons  systems.  Although  a  program  will  be 
considering  systems  of  systems  issues  as  part 
of  their  iterative  development,  there  will  still 
be  macro-level  issues  that  need  to  be  resolved 
at  levels  higher  than  any  one  program.  This 
oversight  function  will  be  able  to  consider  both 
joint  and  combined  systems  of  systems  issues, 
for  those  systems  that  will  operate  with  other 
services  and  allied  nations. 

The  primary  activity  of  the  oversight  function 
is  assessing  and  prioritizing  efforts  from  a 
system  of  systems  perspective.  It  assesses  the 
progress  of  programs’  iterative  developments 
and  the  expected  production  time  to  ensure 
that  the  system  will  be  available  when  required. 
Programs  will  stay  in  iterative  development 
and  continue  to  refine  the  design  in  a  synthetic 
environment,  continually  assessing  new 
technologies  as  well  as  input  from  the  field.  A 
decision  to  begin  production  is  made  in  suffi¬ 
cient  time  to  allow  for  production,  fielding, 
and  training  to  occur  prior  to  the  need  date. 
Oversight  reviews  will  be  periodic  instead  of 
activity-based,  but  can  also  be  unscheduled  in 
response  to  unforeseen  changes  in  the  need. 
These  reviews  will  replace  the  current  mile¬ 
stone  reviews,  because  there  will  be  minimal 
need  for  successive  and  increasing  levels  of 
dollar  commitment  as  the  program  design 
matures  because  the  dollar  commitment  will 
be  significantly  reduced.  Instead,  the  ability 
to  conduct  most  of  the  design  in  the  synthetic 
environment  will  have  the  effect  of  fusing 
the  activities  leading  up  to  today’s  present 
Milestone  III  production  decision. 


Production  Phase 

The  third  major  phase  of  a  program  in  this 
future  state  of  SBA  is  production,  which  begins 
after  a  production  decision  is  made  at  the  end 
of  the  design  and  development  phase.  All 
models  and  simulations  previously  developed 
for  a  program  are  available  for  production. 
These  models  and  simulations  include  tool¬ 
ing  and  factory  design.  Existing  models  and 
simulations  are  developed  to  a  greater  level 
of  detail  as  necessary  for  full-scale  production. 
As  a  result  of  previous  rigorous  development 
and  validation  of  the  models  and  simulations 
there  should  be: 

•  Minimum  Engineering  Change  Proposals 
because  the  design  coming  into  the  Produc¬ 
tion  Phase  was  determined  acceptable  to 
all  stakeholders; 

•  Reduced  tooling  time  because  tooling 
requirements  were  already  identified; 

•  Less  decisions  to  be  made  because  there 
should  be  few  surprises  and  therefore  less 
tradeoff  decisions  required — all  known 
tradeoffs  were  evaluated  and  decided  on; 

•  Better  quality  of  the  end  item  because  reli¬ 
ability,  maintainability,  supportability, 
manufacturability,  producibility,  and  other 
support  issues  were  all  considered  and 
incorporated  into  the  design; 

•  An  affordable  design  because  evaluation 
of  cost  models  with  Cost  As  an  Indepen¬ 
dent  Variable  (CAIV)  forced  early  trade¬ 
off  decisions  between  performance  and 
affordability. 
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Operations  and  Sustainment  Phase 

The  fourth  and  final  major  phase  of  a  program 
in  this  future  state  of  SBA  is  operations  and 
sustainment,  which  ends  upon  system  disposal. 
Actual  operations  and  support  (O&S)  data  are 
used  to  update  the  system’s  models  and  simu¬ 
lations.  There  is  continuous  analysis  of  this 
O&S  feedback  to  determine  future  deficien¬ 
cies.  A  deficiency  may  only  require  a  system 
upgrade  or  it  may  require  development  of  a 
totally  new  concept.  Proposed  changes  to  the 
existing  models  and  simulations  or  new  mod¬ 
els  and  simulations  to  meet  the  deficiency  are 
again  evaluated  in  a  synthetic  environment  and 
the  warfighters  can  use  this  data  to  assess  non¬ 
material  alternatives,  based  upon  the  acquisi¬ 
tion  community’s  updated  models.  Opera¬ 
tional  units  and  commands  could  use  these 
models  and  simulations  to  better  predict  their 
budget  requirements.  Cost  models  that  reflect 
actual  costs  incurred  for  operations  and 
support  of  the  system  could  be  used  to  accu¬ 
rately  project  quarterly  and  annual  training 
and  operational  requirements. 


Net  Result 

This  proactive  approach  to  assessing  program 
status  is  driven  by  mission  need  and  the  date  a 
system  is  required,  rather  than  the  excessive 
length  of  the  acquisition  cycle.  Programs  can 
be  much  more  flexible  to  respond  to  changing 
mission  needs,  because  there  is  not  as  much 
emphasis  on  locking  down  the  requirements. 
The  entire  process  is  more  agile  because  pro¬ 
grams  are  able  to  stay  in  iterative  development 
until  production  lead  times  coincide  with  need 
dates. 

The  ultimate  effect  of  this  new  SBA  process 
will  be  the  ability  to  conduct  “what  if” 
scenarios  across  the  entire  spectrum  of  the 
DoD.  Program  teams  will  be  charged  with 
evaluating  the  interactions  of  their  system(s) 
with  other  systems  on  the  battlefield.  The  over¬ 
sight  function  will  be  armed  with  the  knowl¬ 
edge  to  make  better  informed  decisions  on 
which  systems  will  best  meet  the  military’s 
needs.  Program  teams  will  have  a  much  closer 
link  to  the  battlefield  upon  which  the  systems 
operate,  because  they  will  be  able  to  constantly 
see  the  battlefield  effects  and  implications  of 
their  design  decisions.  All  will  help  keep  the 
team  focused  on  the  real  goal  of  producing 
superior  systems. 
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1.  LTG  George  Muellner,  Air  Force  Principal  3.  CJCSI  3170.01,  Requirements  Generation 

Deputy  for  Acquisition,  interview  by  authors,  30  System  (Formerly  MOP  77)),  13  June  1997. 

March  1998. 

2.  For  a  more  complete  description  of  this  alterna¬ 
tive  view,  see  “Functional  Description  Document 
-  980209”  developed  by  the  NDIA’s  Industry 
Steering  Group,  available  through  the  SBA  Spe¬ 
cial  Interest  Area  on-line  http://www.msosa.mil. 
inter.net/sia-sba/sba_sia_documents.asp? 
process = view_archive 


6-7 


6-8 


CHALLENGES  TO 
IMPLEMENTING  SBA 


Many  programs  are  making  great  progress  in 
implementing  SBA;  however,  even  the  best 
programs  are  a  long  way  from  implementing 
the  ideal  process  outlined  in  the  previous  chap¬ 
ter.  This  chapter  outlines  the  cultural,  techni¬ 
cal,  and  procedural  challenges  to  implement¬ 
ing  this  future  state  of  SBA.  Some  of  these 
challenges  will  be  overcome  more  quickly  than 
others  depending  to  a  large  extent  on  how 
much  emphasis  they  receive.  Cultural  chal¬ 
lenges  result  from  people’s  views,  beliefs,  and 
management  actions;  technical  challenges 
result  from  M&S  limitations  in  technology; 
and  process  challenges  are  caused  by  the  way 
the  DoD  is  organized  and  operates. 

Cultural  Challenges 

Many  people  maintain  that  cultural  barriers 
are  the  biggest  challenge  to  overcome  in  mov¬ 
ing  to  an  SBA  process.  These  challenges 
include  acceptance  of  M&S,  not  believing 
everything  seen,  working  as  teams  in  a 
distributed  environment,  and  becoming 
proactive  in  determining  an  affordable  design. 

Acceptance  of  the  results  of  M&S  is  probably 
the  single  biggest  challenge,  as  it  involves  a 


significant  change  in  the  way  DoD  does 
business.  There  is  a  natural  resistance  to 
changing  the  old  ways  if  they  appear  to  be 
working.  There  is  significant  resistance  to 
believing  in  virtual  prototypes,  as  people  are 
more  apt  to  believe  one  physical  test  versus 
thousands  of  virtual  tests.  For  example,  the 
technology  for  simulating  the  crashworthiness 
of  a  vehicle  far  outpaces  the  acceptance  of  the 
results,  to  the  point  where  their  accuracy  is 
better  than  the  variability  between  different 
physical  crashes.^  However,  despite  this  capa¬ 
bility,  all  automobile  manufacturers  continue 
to  conduct  expensive  crash  tests  (albeit  they 
are  conducting  much  fewer  tests  than  in  the 
past).  A  culture  change  is  required  to  view 
computer  modeling  as  analogous  to  physical 
modeling.  Even  so,  acceptance  of  M&S  is 
occurring  at  an  increasing  rate  as  M&S  is 
successfully  implemented.  The  engineers 
frequently  resist  using  M&S  at  first,  but  soon 
discover  that  it  can  be  a  very  valuable  design 
tool,  and  that  they  often  do  not  need  hardware 
to  find  problems. 

An  important  corollary  to  this  acceptance 
problem  is  to  “not  believe  everything  you 
see.”  Simulations  create  compelling  visual 


arguments  that  can  drive  complacency  in  ques¬ 
tioning  the  underlying  models.  The  mere  fact 
that  something  can  be  modeled  and  simulated 
does  not  mean  that  it  can  be  built,  A  rigorous 
process  of  verifying,  validating,  and  accredit¬ 
ing  (VV&A)  models  and  simulations  for  their 
intended  purpose  can  counteract  this  problem, 
and  can  increase  the  level  of  confidence  that 
the  models  and  simulations  are  representative 
of  what  was  intended  and  accurately  represent 
reality.^ 

Another  cultural  challenge  is  learning  how  to 
work  as  teams  in  distributed  environments. 
Western  culture  and  training  places  an  empha¬ 
sis  on  individual  performance,  but  systems  are 
designed  and  built  by  teams.  An  SBA  process 
can  accentuate  this  long-standing  problem 
because  even  more  emphasis  is  placed  on 
working  together  as  a  team,  and  the  teams  may 
change  frequently  and  not  be  co-located. 
Many  companies  are  experimenting  with  ways 
to  create  just-in-time  teams  and  reward  team 
performance. 

A  final  cultural  challenge  is  learning  to  become 
proactive  in  designing  an  affordable  system. 
This  involves  continually  refining  the  require¬ 
ments  with  the  user  to  find  the  optimum  mix 
of  performance  and  affordability,  and  look¬ 
ing  at  the  costs  across  the  entire  life  of  the 
system.  The  range  of  options  to  explore  can 
be  intimidating,  and  the  tendency  will  be  to 
move  on  with  a  design  that  appears  good 
enough,  rather  than  continuing  to  explore 
design  options  to  reduce  the  TOC  of  the 
system. 

Technical  Challenges 

Technical  barriers  are  perhaps  the  easiest  to 
envision.  Technical  challenges  include 
interoperability  concerns,  and  limitations  in 


what  can  be  effectively  and  affordably 
modeled. 

Interoperability  challenges  result  from  the 
desired  ability  to  seamlessly  share  models  and 
simulations  between  programs  and  across 
services  to  analyze  tradeoffs  in  a  system  of 
systems  synthetic  environment.  As  discussed 
previously,  the  DMSO  is  pursuing  various 
initiatives  in  the  Common  Technical  Frame¬ 
work  to  facilitate  interoperability  and  reuse. 
Certainly  more  technical  improvements  will 
continue  emerging,  such  as  computer-aided 
tools,  which  will  facilitate  the  development  and 
deployment  of  reuse  standards  and  policies 
such  as  the  High  Level  Architecture. 

Programs  can  experience  interoperability  chal¬ 
lenges  within  their  own  program,  as  they 
attempt  to  move  information  up  and  down  the 
hierarchy  of  models  and  simulations  (for 
example,  from  campaign  to  mission  to 
engagement  to  engineering,  and  then  back 
up).  There  are  interoperability  challenges  in 
making  different  tools  communicate  with  each 
other,  particularly  those  that  were  developed 
for  use  in  a  different  context  from  the  one  for 
which  they  are  now  intended.  For  example,  a 
logistics  model  that  helps  determine  the 
optimum  stockage  level  for  spare  parts  may 
not  be  compatible  with  another  tool  that 
predicts  reliability,  or  even  worse,  gives 
conflicting  results. 

The  MSRR  will  provide  increasing  access  to 
programs  to  assist  in  their  finding  what  models 
and  simulations  are  available  for  their  systems 
of  systems  analyses,  which  will  facilitate  maxi¬ 
mum  reuse.  This  is  particularly  evident  with 
items  that  have  been  designed  for  reuse,  such 
as  threat  and  environment  data.  However, 
designing  for  reuse  costs  approximately  three 
times  as  much  as  designing  for  a  specific 


7-2 


program,^  and  is  only  feasible  when  all  of  the 
programs  intending  to  use  the  models  and 
simulations  are  known  in  advance.  HLA  be¬ 
gins  to  solve  this  problem  by  helping  groups 
that  choose  to  interact  with  each  other  define 
specific  system  architectures  called  federa¬ 
tions.  This  creates  a  problem  when  a  program 
wants  to  use  another  program’s  model  or  simu¬ 
lation  that  was  developed  as  part  of  another 
federation.  An  extreme  but  probable  example 
would  be  attempting  to  use  models  and  simu¬ 
lations  from  several  other  federations  to  get  a 
true  picture  of  the  battlefield. 

In  an  ideal  SBA  process,  models  and  simula¬ 
tions  can  be  reused  as  easily  as  using  a  library 
or  repository.  However,  historically  this  type 
of  ad  hoc  reuse  has  only  resulted  in  about  20 
percent  savings  in  software  code  reuse."*  A  lot 
of  time  is  often  spent  analyzing  the  code  to 
determine  what  is  applicable  for  reuse,  and 
often  the  most  valuable  results  are  the  insights 
gained  into  how  to  functionally  design  the  new 
code,  rather  than  the  reuse  of  the  code  itself. 
A  SBA  process  may  not  actually  reuse  models 
and  simulations  in  this  traditional  software 
sense,  but  it  does  point  out  the  considerable 
difficulty  experienced  in  a  similar  process 
which  is  much  simpler  than  SBA  envisions. 

Some  of  the  easiest  areas  of  reuse  to  envision 
are  for  common  areas  that  many  programs  will 
need,  such  as  models  of  the  environment  in 
which  systems  will  operate,  and  of  the  threats 
these  systems  will  face.  Efforts  are  already 
ongoing  in  some  of  these  areas,  such  as  the 
SEDRIS  effort  mentioned  earlier,  and  the  Vir¬ 
tual  Proving  Ground  under  development  at 
the  Army’s  Test  and  Evaluation  Command. 
The  DoD  has  a  vested  interest  in  reducing 
duplication  of  efforts  in  these  areas,  and  also 
in  having  a  common  depiction  of  them  across 
all  systems.  Programs  will  be  able  to  use  the 


DoD’s  environment  to  evaluate  many  aspects 
of  their  system.  If  a  program  has  a  require¬ 
ment  that  is  not  satisfied  by  the  common 
environment,  it  can  be  expanded  to  meet  the 
needs,  and  it  can  be  subsequently  brought  back 
into  the  aggregate  common  environment  for 
reuse  by  other  programs.  The  common  envi¬ 
ronment  gets  expanded  as  a  result.  Having  a 
common  environment  will  also  provide  cred¬ 
ibility  and  help  to  minimize  issues  that  could 
arise  if  each  contractor  developed  his  own 
environment  and  threat  models,  especially  if 
these  systems  were  in  a  competitive  selection 
process.  Other  items  the  DoD  should  consider 
providing  are  the  analytical  tools  that  will  be 
used  to  evaluate  contractors’  models.  For 
example,  if  the  DoD  planned  to  use  a  construc¬ 
tive  model  such  as  the  Army’s  CASTFOREM 
(Combined  Arms  and  Support  Task  Force 
Evaluation  Model)  to  evaluate  a  contractor’s 
proposed  model,  then  the  program  office 
should  consider  providing  CASTFOREM  to 
the  contractor  early  in  the  program. 

There  are  also  technical  limitations  in  what 
can  be  effectively  and  affordably  modeled, 
including  reliable  cost  figures,  failure  and 
reliability  prediction,  and  human  behavior. 
Cost  models  that  accurately  project  TOC  need 
to  be  available  as  early  as  possible  to  enable 
informed  tradeoff  analyses  that  all  communi¬ 
ties  and  decision  makers  can  believe.  Discrep¬ 
ancies  will  otherwise  persist  between  cost 
estimates  prepared  by  differing  agencies — for 
example,  the  Government  Accounting  Office 
might  predict  higher  cost  estimates  than  the 
program’s  estimates.  These  discrepancies 
could  effectively  cripple  the  program’s  ability 
to  design  a  more  affordable  system,  because 
the  credibility  of  the  system’s  total  owner¬ 
ship  costs  that  are  being  used  to  make  de¬ 
sign  tradeoffs  is  called  into  question.  Cost 
curves  for  each  alternative  considered  allow 
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quantification  of  alternatives  and  operational 
tradeoffs  with  cost  as  an  independent  variable. 
The  objective  is  to  use  modeling  and  simula¬ 
tion  to  help  determine  the  “bend  in  the  knee” 
when  looking  at  performance  versus  cost.  This 
bend  in  the  knee  refers  to  the  point  where  the 
cost  of  providing  greater  performance  begins 
to  exceed  the  benefit  it  provides,  or  its  point 
of  marginal  return.  By  showing  the  user  where 
the  bend  in  the  knee  starts  to  occur  on  a  given 
performance  characteristic  then  we  can  ask  if 
it  is  worth  the  extra  cost  to  achieve  the 
incrementally  less  capability  per  dollar  spent. 
The  earlier  we  are  able  to  show  the  user  the 
cost  implications  of  his  requirements,  the 
sooner  we  can  agree  to  the  necessary  tradeoffs 
to  result  in  an  affordable  design.  This  forces 
the  user  to  make  tough  calls  early  in  the 
system’s  development  and  avoids  pursuing 
unaffordable  solutions. 

We  need  to  be  able  to  treat  CAIV  at  the  earli¬ 
est  stages,  by  incorporating  cost  considerations 
into  the  computer-aided  design  and 
engineering  tools.  It  is  otherwise  impossible 
to  get  cost  on  the  table  as  a  trade  mechanism. 
We  are  still  using  cost  models  that  use  dollars 
per  pound  as  the  basis  of  estimating,  as 
opposed  to  conducting  detailed  cost  tradeoffs. 
Just  as  computer-numerically-controlled 
(CNC)  machines  are  able  to  manufacture 
items  directly  from  the  digital  design  data,  cost 
models  would  be  able  to  directly  use  the  digital 
data  to  project  manufacturing  costs.  Even 
more  sophisticated  cost  models  would  be  able 
to  address  costs  across  the  life  cycle  of  the 
system — operations  costs,  upgrade  costs, 
maintenance  costs,  and  disposal  costs,  for 
example. 

Today’s  state-of-the-art  in  modeling  and  simu¬ 
lation  does  not  provide  the  ability  to  fully  and 
accurately  predict  failures  to  the  point  where 


we  can  replace  physical  reliability  testing  with 
reliability  testing  using  modeling  and  simula¬ 
tion.  Durability  algorithms  are  in  short  sup¬ 
ply,  because  of  the  complexity  of  durability 
models.  The  ability  to  predict  failure  varies 
from  one  product  area  to  another.  Physics  of 
failure  is  farthest  along  in  electronics.^  For 
example,  transistor  manufacturers  are  able  to 
predict  with  high  confidence  when  transistors 
will  fail.  In  chips,  we  know  exactly  how  they’ll 
function  and  how  they’ll  behave,  but  the  same 
does  not  hold  for  the  mechanical  world.  Cur¬ 
rently,  we  do  not  have  good  models  for  pre¬ 
dicting  failure  and  determining  reliability  of 
mechanical  systems  or  their  components.  The 
ability  to  predict  the  failure  of  a  new  system 
requires  an  understanding  of  the  internal  phys¬ 
ics  of  the  materials  within  the  system.  The 
ability  to  model  this  level  of  fidelity  within 
components  and  within  a  system  is  in  its 
infancy,  but  some  fields  of  study  are  making 
strides  by  applying  recent  increases  in  com¬ 
puting  power.  Meteorological  experts  are 
beginning  to  model  weather  systems  at 
increasing  levels  of  fidelity  to  achieve  a  better 
understanding  of  how  thunderstorms,  hurri¬ 
canes,  and  tornadoes  behave.  Pharmaceutical 
companies  are  beginning  to  model  new  drugs 
at  the  atomic  level  and  then  immerse  them¬ 
selves  into  the  model  to  help  them  create  the 
drug.  We  can  expect  similar  advances  in  other 
fields  as  the  technology  matures.  We  need  to 
develop  physics  of  failure  and  cost  models  to 
help  predict  reliability  through  simulation.  If, 
in  simulation,  we  are  able  to  test  a  program  to 
the  point  of  failure,  then  we  can  improve  over¬ 
all  system  reliability  by  improving  the  area  that 
failed.  We  can  also  use  these  failure  data  to 
predict  the  required  maintenance,  manpower, 
spares,  and  life  cycle  cost  for  a  system  given 
the  number  of  operating  hours. 
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The  limitations  to  our  ability  to  build  high 
fidelity  physics  of  failure  models  result  from 
such  things  as  the  limited  understanding  of 
how  materials  fail,  variations  in  the  manufac¬ 
turing  processes  of  materials,  and  variations 
in  the  properties  of  materials.  One  of  the  most 
significant  limitations  is  the  cost  and  time 
required  to  capture  all  the  stress  loads  that  a 
system  or  component  will  experience  during 
operations.  Each  test  conducted  on  a  system 
captures  data  for  only  one  set  of  operational 
conditions.  Program  schedule  and  cost  con¬ 
straints  limit  the  number  of  samples  that  can 
be  taken,  so  assumptions  are  necessary  to 
extrapolate  from  this  limited  sample  size  to 
make  conclusions  for  all  possible  conditions 
that  may  be  experienced.  If  we  could  accu¬ 
rately  model  the  physics  of  the  system,  we 
could  conduct  many  more  tests  in  the  synthetic 
environment  than  are  cost-effective  with 
today’s  approach. 

A  program  manager  must  demonstrate  that 
the  system  he  is  developing  will  meet  the  reli¬ 
ability  required  by  the  user.  Durability  testing 
is  the  process  that  provides  the  evidence  of 
whether  a  system  is  reliable  enough  to  pro¬ 
ceed  toward  production.  Durability  testing  is 
very  demanding  of  program  resources, 
requiring  the  PM  to  schedule  and  pay  for  hun¬ 
dreds,  if  not  thousands  of  hours  of  system 
operations  and  testing.  Confidence  in  the 
reliability  of  the  systems  that  we  field  requires 
that  we  have  an  acceptable  level  of  under¬ 
standing  about  why  and  how  often  a  system 
will  fail,  which  can  be  a  significant  cost  driver 
of  a  program.^  The  challenge  is  to  reduce  the 
number  of  reliability  testing  hours  required. 
If  we  can  reduce  the  number  of  reliability  test¬ 
ing  hours  required  of  an  end  item,  we  can 
reduce  both  the  schedule  and  budget  neces¬ 
sary  for  a  program  during  this  phase.  As  an 
example,  the  M1A2  tank  program  dedicated 


four  physical  prototype  vehicles  just  to  support 
durability  testing.'^ 

Today,  when  we  develop  a  new  system  it  has 
no  known  reliability.^  Only  through  use  and 
time  do  we  achieve  reliability.  The  only  true 
way  to  achieve  this  knowledge  is  to  run  the 
physical  system.  When  we  do  this  we  gain  con¬ 
fidence  in  the  reliability  of  that  one  system  we 
tested  and  then  we  infer  reliability  to  any  sys¬ 
tem  that  is  built  to  the  same  specifications. 
Therefore,  the  two  critical  factors  are  first,  that 
the  prototype  we  test  be  in  accordance  with 
the  system’s  specifications;  and  second,  that 
we  are  able  to  accurately  repeat  builds  of  sub¬ 
sequent  systems.  The  weakness  of  this  ap¬ 
proach  is  that  the  physical  prototype  is  likely 
to  have  flaws  and  inconsistencies  compared 
with  the  final  production  version  of  the  sys¬ 
tem  because  of  human  error  in  building  the 
prototype,  and  configuration  changes  as  we 
gain  more  knowledge  about  the  system  and  as 
technology  continues  to  develop. 

Modeling  and  simulation  actually  enables  us 
to  improve  the  confidence  in  the  ability  to 
accurately  build  and  test  a  prototype  that  is 
truly  representative  of  a  production  item. 
Using  M&S  we  are  able  to  keep  the  design 
open  to  changes  longer  because  we  have 
greater  confidence  that  the  final  design  will 
meet  the  required  need  and  that  the  produc¬ 
tion  run  will  be  successful.  This  means  we  need 
less  time  allocated  for  last  minute  changes  and 
gives  us  more  time  to  learn  about  the  system 
as  it  matures,  to  incorporate  lessons  learned 
back  into  the  system,  and  to  integrate  the  latest 
technology  if  desired.  The  changes  made  using 
M&S  early  in  the  design  can  be  much  less 
expensive  than  if  we  were  making  changes  to 
a  physical  prototype,  which  would  not  be 
possible  until  much  later  in  the  program. 
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A  final  technical  limitation  is  human  behav¬ 
ior  modeling.  Conceptually,  SBA  could 
include  modeling  and  simulating  the  acquisi¬ 
tion  process  itself  to  help  produce  better 
acquisition  strategies.  Human  behavior  is  dif¬ 
ficult  to  model,  however,  because  our 
assumptions  cannot  be  correct  for  all  individu¬ 
als.  Some  factors  that  influence  how  an  indi¬ 
vidual  will  react  in  a  given  situation  include 
experience,  education,  values,  self-confidence, 
and  fatigue.  The  variability  in  reactions  is  so 
great  that  accurately  predicting  behavior  is  not 
possible  and  any  process,  such  as  acquisition, 
that  is  heavily  dependent  on  human  interface 
and  decisions  is  difficult  to  predict  and  model. 
Probably  the  best  we’ll  be  able  to  achieve  in 
this  area  will  be  through  the  use  of  wargaming 
techniques.  Wargaming  allows  for  shortfalls  in 
models  and  data,  and  provides  a  way  for  pro¬ 
gram  managers  to  simulate  interactions  with 
entities  that  cannot  be  directly  controlled.  The 
insights  gained  by  understanding  the 
implications  of  decisions  should  enable 
managers  to  develop  more  effective 
strategies.^ 

Process  Challenges 

Process  challenges  result  from  the  way  DoD 
operates  and  is  organized,  and  include  the 
change  in  the  test  process,  security  of  classified 
and  proprietary  data,  and  metrics. 

A  major  concern  of  program  managers  is  the 
cost  and  time  required  to  test  a  system.  A 
significant  portion  of  a  program’s  schedule 
and  budget  is  usually  allocated  for  testing, 
and  an  SBA  approach  has  the  potential  of 
reducing  both.  As  mentioned  earlier,  the 
role  of  testing  is  no  longer  to  validate  point 
designs;  rather  the  role  of  testing  is  to  vali¬ 
date  models  so  that  there  is  a  high  confidence 
in  the  resultant  simulations  from  those  models. 


Test  personnel  should  be  members  of  the  IPT 
during  the  earliest  stages,  and  should  work 
with  the  developer  to  consider  and  resolve  test 
issues.  A  good  example  of  addressing  test 
issues  early  is  the  Follow  On  To  TOW  (Tube- 
Launched,  Optically  Tracked,  Wire  Guided) 
hand-held  anti-tank  missile,  or  FOTT.  In  the 
FOTT  program  the  specifications  for  the  Vir¬ 
tual  Proving  Ground  were  included  in  the 
Request  for  Proposal,  and  test  personnel  were 
co-located  with  the  program  developers.^® 
These  test  personnel  grew  up  with  the  models 
and  matured  them  with  tests  as  required  to 
gain  confidence  in  their  use.  They  also  helped 
develop  the  acceptance  test  procedures  (ATP), 
so  that  many  iterations  and  excursions  could 
occur  quickly  and  inexpensively  and  the  results 
could  be  checked  against  the  ATP  after  each 
iteration.  Involving  developmental  test  and 
operational  test  personnel  early  in  the  pro¬ 
gram  and  using  M&S  to  gain  higher  confi¬ 
dence  in  the  first  missile  allowed  the  FOTT 
PM  to  reduce  missile  requirements  for  engi¬ 
neering  and  manufacturing  development 
(EMD).  The  Javelin  program  (a  similar  fire- 
and-forget  missile)  required  190  missiles  for 
EMD,  while  FOTT  required  only  40  missiles.^^ 

Within  the  test  community  there  needs  to  be 
a  shift  in  focus  from  testing  hardware  to  test¬ 
ing  simulations  of  hardware.  By  testing  simu¬ 
lations  of  future  hardware  systems,  IPT  mem¬ 
bers  can  resolve  many  problems  before  expen¬ 
sive  prototypes  are  built.  Test  simulations  help 
to  identify  the  problem  areas  where  efforts 
should  be  focused  to  increase  the  probability 
that  live  testing  is  successful.  The  knowledge 
gained  from  conducting  these  simulations  will 
lead  to  better  quality  testing  as  well  as  more 
efficient  and  less  testing  of  the  physical  hard¬ 
ware.  As  an  example,  during  the  Longbow 
helicopter  Hellfire  missile  development  there 
were  more  than  500  simulated  firings  and  only 
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20  live  shots.  This  resulted  in  the  Longbow 
Simulation/Test  Acceptance  Facility  (STAF) 
paying  for  itself  in  one  year,  with  several  years 
left  in  the  program.^^  In  addition,  other 
programs  will  be  able  to  “reuse”  the  facility, 
further  amplifying  the  cost  avoidance  for  DoD. 

The  purpose  of  conducting  live  testing  is 
changing  from  validation  of  the  physical  sys¬ 
tem  to  validation  of  the  model.  General 
Motors  is  using  tests  to  validate  model  design 
assumptions  and  not  to  find  fixes.  The  goal  is 
to  get  the  design  right  as  early  in  the  life  qrcle 
as  possible.  As  confidence  in  the  results  of 
models  goes  up  the  number  of  live  tests 
required  will  come  down.  Any  questions  that 
simulations  cannot  answer  may  require  a 
physical  prototype  of  the  subsystem  in  ques¬ 
tion  to  reduce  risk  to  an  acceptable  level. 
Information  gained  from  building  and  testing 
any  physical  prototype  of  the  system  should 
be  used  to  update  the  system’s  model. 

A  big  challenge  is  to  have  an  authoritative 
DoD  source  for  a  common  synthetic  test 
environment.  If  we  intend  to  compare  com¬ 
peting  system  designs,  we  need  to  ensure  we 
evaluate  those  systems  within  a  common 
synthetic  environment  that  is  also  available  to 
the  system  developers.  This  common 
environment  is  required  to  provide  a  level 
playing  field  for  evaluating  conceptual  designs 
of  a  system.  These  relationships  are  critical  in 
ensuring  that  the  run-time  databases,  derived 
from  the  synthetic  environment  database,  will 
be  correlated  so  that  all  “views”  of  the  envi¬ 
ronment  are  the  same.  An  important  “view” 
is  that  of  computer-generated  forces  that  do 
not  “see”  the  battlefield  but  must  use  the  data 
representations  to  correctly  interpret  the 
environmental  conditions. This  means  that 
the  computer  controlling  the  actions  of  mili¬ 
tary  forces  in  a  simulation  needs  information 


about  the  terrain  and  objects  in  the  environ¬ 
ment  represented  in  a  way  it  can  interpret.  For 
example,  the  visual  representation  of  an  object 
that  a  soldier-in-the-loop  needs  during  a  simu¬ 
lation  is  no  good  to  the  computer.  The  com¬ 
puter  needs  information  about  the  same  ter¬ 
rain  and  objects,  but  numerically  represents 
the  essential  information,  such  as  position, 
height,  weight,  etc.  Unfortunately,  synthetic 
environment  data  interchange  today  is  cum¬ 
bersome,  expensive,  and  often  unreliable. 

A  second  major  process  concern  among 
government  and  industry  is  ensuring  the 
security  of  classified  and  proprietary  data.  If 
we  expect  agencies  and  companies  to  provide 
data  through  repositories,  we  must  be  able  to 
provide  a  high  level  of  confidence  in  the  secu¬ 
rity  of  those  data.  We  want  data  to  be  readily 
available  and  shared  with  those  who  need 
the  information  and  at  the  same  time,  we 
must  keep  data  from  those  who  do  not  have 
a  need  to  know.  Today’s  technology  allows 
us  to  share  data  easily  and  there  are  many 
ways  that  data  can  get  into  the  wrong  hands. 
As  an  example,  an  automobile  manufacturer 
inadvertently  left  some  data  of  a  “next  gen¬ 
eration”  vehicle  on  a  data  tape  that  was 
reused  and  given  to  one  of  its  suppliers.  The 
supplier  inadvertently  passed  the  data  on  to 
a  competitor.  This  security  breach  of  cor¬ 
porate  knowledge  allowed  the  competition 
to  make  up  a  year-and-a-half  in  development 
time.^^ 

Program  offices  are  also  very  careful  about 
who  uses  the  models  and  simulations  of  their 
program.  They  want  to  know  exactly  how 
and  for  what  purpose  the  data  will  be  used. 
This  is  understandable  because  a  compet¬ 
ing  program  could  use  the  detailed  infor¬ 
mation  provided  by  the  models  and  simula¬ 
tions  of  another  program  to  show  how  its 
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system  is  superior  or  where  the  other  pro¬ 
gram  is  deficient.  The  information  contained 
in  a  model  could  also  be  a  valuable  source  to 
determine  vulnerabilities  of  the  new  system. 

A  final  process  challenge  will  be  developing 
metrics,  which  are  needed  for  an  overall 
assessment  of  progress  in  the  execution  of  a 
program.  Metrics  are  measures  of  success  that 
serve  as  a  powerful  management  tool  for 
evaluating  effectiveness  in  accomplishing 
project  objectives  and  in  achieving  and 
improving  customer  satisfaction,^^  They  allow 
program  managers  to  manage  on  the  basis  of 
facts  and  data.  Metrics  solely  focused  on 
individual  process  results  do  not  give  a  picture 
of  overall  success  in  implementation.  Metrics, 
therefore,  should  also  be  structured  to  identify 
the  overall  effects  of  SBA  implementation. 
Measures  that  could  be  used  to  gauge  suc¬ 
cess  include  schedule,  responsiveness  and 
timeliness,  and  communications.  The  mea¬ 
surement  of  variances  between  planned  and 
actual  schedules,  consumption  of  resources, 


productivity,  customer  satisfaction,  cycle  time, 
and  completion  of  tasks  could  be  particularly 
useful  in  determining  if  a  program  is  captur¬ 
ing  the  expected  benefits  and  cost  avoidance. 
In  a  collaborative,  concurrent,  and  distributed 
engineering  environment,  communications 
takes  on  an  increasingly  important  role. 
Organizations  must  be  able  to  easily  and 
quickly  move  large  amounts  of  data.  They 
must  track  how  well  information  is  moving 
through  the  organization,  such  as  whether  the 
Product  Information  Manager  is  facilitating 
concurrent  engineering,  or  whether  the  engi¬ 
neers  are  experiencing  unacceptable  delay 
times. 

There  are  difficult  cultural,  technical,  and  pro¬ 
cedural  challenges  to  implementing  the  ideal 
SBA  process  of  the  future.  But  we  don’t  have 
to  wait  until  all  the  challenges  are  solved  be¬ 
fore  we  start  implementing  those  parts  where 
we  can  extract  value.  The  process  is  beginning 
to  change,  and  those  who  adapt  will  find  the 
biggest  rewards. 
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CONCLUSIONS  AND 
RECOMMENDATION  S 


This  chapter  provides  our  conclusions  regard¬ 
ing  SBA,  along  with  some  recommendations 
on  where  to  proceed.  Our  conclusions  are  that 
SBA  is  a  smarter  way  of  doing  business,  but 
there  are  significant  challenges  ahead  to  real¬ 
izing  its  full  potential.  Our  recommendations 
are: 

•  educate  the  workforce  on  this  new  concept; 

•  encourage  a  single  SBA  process  for  DoD; 

•  identify  SBA  efforts  within  program 
documents; 

•  continue  developing  the  tools  and  infra¬ 
structure  to  make  it  happen; 

•  fund  SBA  as  well  as  M&S  development; 
and 

•  conduct  a  follow-on  study  that  results  in  an 
SBA  Recommended  Practices  Guide. 

Conclusions 

SBA  is  a  smarter  way  of  doing  business.  It 
creates  an  environment  where  complex  issues 
can  be  integrated  and  evaluated  to  support 


decision  makers  throughout  the  life  cycle  of  a 
program.  Risks  can  be  identified,  minimized, 
mitigated,  or  even  eliminated  to  increase  the 
probability  of  program  success.  Products 
developed  when  implementing  SBA  can  serve 
as  excellent  communication  and  marketing 
tools.  Being  able  to  visualize  a  system  as  it 
evolves  from  a  low  fidelity  concept  into  a  high 
fidelity  system  is  very  useful  in  resolving  con¬ 
flicts  throughout  development.  Simulation  and 
visualization  are  powerful  tools  for  IPTs  and 
decision  makers  which  allow  them  to  see  the 
results  of  each  decision  and  evaluate  the  rami¬ 
fications  before  having  to  make  a  costly  final 
decision. 

Using  SBA,  programs  are  making  decisions 
based  not  only  on  more  information,  but  also 
better  information,  which  is  leading  to  a  better 
product.  These  programs  are  being  “pushed” 
into  using  SBA  for  a  variety  of  reasons,  but 
once  there,  they  are  finding  many  benefits. 
Both  industry  and  government  programs  are 
turning  to  SBA  as  a  means  of  remaining  com¬ 
petitive  and  making  the  most  of  limited  funds. 
Companies  that  have  implemented  SBA  on 
one  program  have  started  to  apply  SBA  to 
other  programs  and  are  looking  for  ways  to 
leverage  efforts  across  programs.  Many 


programs  are  discovering  that  using  M&S  is 
the  only  way  to  accomplish  certain  tasks  be¬ 
cause  of  cost,  safety,  time,  or  scale  consider¬ 
ations.  As  decision  makers  and  users  begin  to 
see  programs  implementing  SBA,  there  will 
be  a  “pull,”  or  an  expectation  that  SBA  will 
be  used  in  other  programs.  As  they  are  shown 
the  cost  impacts,  warfighters  are  beginning  to 
see  an  impact  and  the  value  of  delaying  the 
finalizing  of  their  requirements.  Programs  us¬ 
ing  SBA  will  be  perceived  as  better  and  in  fact 
will  be  better,  than  programs  not  using  SBA. 
SBA  will  enable  programs  to  quickly  deter¬ 
mine  what  the  issues  are  and  to  focus  on  them. 
Not  only  will  the  program  benefit  from  imple¬ 
menting  an  SBA  approach,  but  outside  orga¬ 
nizations  will  benefit  also. 

The  roles  of  industry  and  DoD  are  unclear  for 
implementing  SBA,  however.  There  is 
uncertainty  about  who  should  make  the  first 
move  toward  institutionalizing  SBA.  In  the 
spirit  of  the  Single  Process  Initiative  to  follow 
best  commercial  practices,  DoD  is  reluctant 
to  dictate  to  industry  how  to  implement  SBA, 
believing  that  it  will  occur  naturally  in  the  com¬ 
petitive  commercial  world.  Industry’s  attitude, 
on  the  other  hand,  is  that  if  DoD  wants  a  stan¬ 
dard  system,  it  will  have  to  put  some  money 
behind  development.^  Industry  is  particularly 
concerned  that  each  service  will  come  up  with 
different  standards  and  ways  to  implement 
SBA. 

As  discussed  in  the  last  chapter,  there  are  tech¬ 
nical,  process,  and  cultural  challenges  to 
implementing  SBA.  The  technical  challenges 
are  impeding  the  full  potential  of  SBA,  but 
are  continuing  to  improve  at  various  paces 
among  the  domains.  SBA  will  continue  to  pick 
up  speed  and  move  toward  a  critical  mass  as 
these  barriers  continue  to  fall.  The  process 
challenges  require  further  analysis  and  study. 


SBA  provides  a  significantly  different 
approach  to  identifying  and  meeting  military 
requirements  from  the  current  process.  The 
implications  of  a  collaborative,  concurrent 
development  approach  versus  the  present 
sequential,  increasing  buy-in  approach  are  not 
well  understood,  and  may  require  us  to  re¬ 
evaluate  our  current  process.  Activities  within 
the  process  may  need  to  occur  more  often  or 
at  a  different  time  in  the  development  of  a 
system.  The  cultural  challenges  require 
education  and  experience — education  to 
spread  the  word  on  this  new  perspective  on  us¬ 
ing  M&S,  and  experience  that  comes  by  getting 
started  and  learning  with  a  new  process.  Many 
programs  are  doing  smart  things  and  there  is  a 
great  need  for  educating  all  SBA  stakeholders 
about  what  SBA  is  and  how  it  is  different  from 
just  using  M&S  in  a  program.  Acceptance  of 
SBA  will  continue  to  grow  as  more  people 
learn  its  concept,  capabilities,  and  benefits. 
Gaining  experience  with  SBA  is  one  of  the 
best  ways  of  overcoming  cultural  barriers. 

Recommendations 

Our  first  recommendation  is  to  educate  the 
workforce  on  the  new  SBA  concept  by  incor¬ 
porating  classes  on  SBA  into  the  curriculum 
at  the  Defense  Systems  Management  College 
that  define  what  SBA  is,  what  tools  are  avail¬ 
able,  what  SBA  efforts  are  ongoing,  and  what 
the  vision  of  SBA  is.  These  classes  should  also 
include  lessons  learned  by  programs  that  suc¬ 
cessfully  implemented  SBA  as  well  as  from 
programs  that  failed  to  implement  SBA.  As 
noted  earlier,  there  is  only  a  single  reference 
to  SBA  in  the  Defense  Acquisition  Deskbook. 
Some  of  the  areas  that  are  now  devoted  to 
M&S  should  be  reviewed  with  the  goal  of 
incorporating  the  broader  viewpoint  of  using 
an  SBA  approach  in  acquisition,  rather  than 
just  the  use  of  M&S.  Once  this  happens,  the 


8-2 


field  should  be  encouraged  to  provide  input 
on  best  practices,  and  wisdom  and  advice.  To 
get  the  dialogue  started,  we  recommend  that 
this  book  be  included  in  the  Deskbook  as  a 
reference  source  for  SBA. 

Our  second  recommendation  is  to  encourage 
all  stakeholders  to  define  and  develop  a  single 
DoD  SBA  process.  The  Services  need  to  work 
together  to  develop  an  agreed  upon  single 
process  for  SBA.  A  single  process  will  reduce 
the  burden  on  contractors  when  dealing  with 
multiple  Services.  The  result  should  be  a 
reduction  in  the  overall  cost  to  DoD  across 
programs.  Certainly  we  don’t  want  a  situation 
that  replicates  the  one  in  some  industries  that 
dictate  specific  software  packages  for  their 
suppliers. 

Our  third  recommendation  is  to  identify  SBA 
efforts  within  existing  program  documents, 
such  as  the  Simulation  Support  Plan,  the  Test 
and  Evaluation  Master  Plan,  and  the  Single 
Acquisition  Management  Plan.  The  intent  is 
to  move  beyond  just  specifying  how  M&S  will 
be  used  in  a  program,  to  outlining  how  M&S 
will  be  applied  to  change  the  process  to  an 
SBA  approach. 

Our  fourth  recommendation  is  to  continue 
developing  the  tools  and  infrastructure  to 
enable  programs  to  conduct  systems  of  systems 
analyses.  The  DoD  should  continue  to  look 
for  investments  in  common  SBA  tools  that  will 
support  analysis  of  programs  within  a  system 
of  systems  synthetic  environment.  DoD 
needs  to  find  the  resources  to  develop  com¬ 
mon  portions  of  the  SBA  infrastructure,  as 
this  issue  is  too  big  and  too  important  to 


leave  to  a  piecemeal  approach  by  individual 
programs,  which  cannot  afford  to  fund  large- 
scale  M&S  efforts  to  benefit  all  programs.  Pro¬ 
grams  do  only  what  is  required  to  successfully 
field  a  system.  A  program  may  develop  a 
capability  that  all  may  benefit  from,  but  this 
is  secondary  and  not  the  focus  of  the  pro¬ 
gram.  Common  tools  developed  by  a  program 
or  by  DoD  need  to  be  available  to  other 
programs. 

Our  fifth  recommendation  is  to  fund  SBA 
development  in  addition  to  M&S  development 
activities.  Efforts  need  to  focus  on  achieving 
the  holistic  view  of  SBA  and  not  on  individual 
and  unrelated  M&S  activities.  The  bigger  pic¬ 
ture  of  SBA  must  be  emphasized  and  efforts 
coordinated  to  optimize  their  contribution  to 
achieving  SBA.  When  screening  priorities  for 
funding,  use  an  SBA  filter  as  well  as  an  M&S 
filter. 

Our  sixth  and  final  recommendation  is  to  con¬ 
duct  a  follow-on  study  to  analyze  the  impacts 
of  SBA  on  cost,  schedule,  and  performance, 
and  from  that  develop  an  SBA  Recommended 
Practices  Guide.  While  we  concentrated  our 
efforts  on  the  success  stories  and  programs 
that  touted  prowess  in  implementing  SBA, 
there  are  probably  failures  and  lessons  learned 
the  hard  way.  Some  of  these  may  be  the  result 
of  technology  limitations,  while  others  are 
errors  in  implementation,  but  they  are  all  valu¬ 
able  lessons.  This  effort  should  focus  on  the 
hard  evidence  to  show  why  SBA  is  indeed  a 
better,  faster,  and  cheaper  way  of  doing  busi¬ 
ness,  and  thereby  ferret  out  the  key  criteria 
programs  should  use  in  determining  how  to 
implement  SBA  in  their  program. 
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APPENDIX  A 

ACRONYMS  AND  GLOSSARY 

ACRONYMS 


Acronym  Term 


AAAV 

ACAT 

ACTD 

ADS 

AI 

APMC 

C4ISR 

CAD 
CAE#1 
CAE  #2 
CAIV 
CAM 
CASTFOREM 
CEP 
CFD 
C4I 
CMMS 
CNC 
CODE 
CRD 
CTF 
DIF 
DMSO 


Advanced  Amphibious  Assault  Vehicle 
Acquisition  Category 

Advanced  Concept  Technology  Demonstration 
Authoritative  Data  Sources 
Artificial  Intelligence 
Advanced  Program  Managers  Course 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance, 
and  Reconnaissance 

Computer  Aided  Design 
Component  Acquisition  Executive 
Computer  Aided  Engineering 
Cost  as  an  Independent  Variable 
Computer  Aided  Manufacturing 

Combined  Arms  and  Support  Task  Force  Evaluation  Model 
Concept  Evaluation  Program 
Computational  Fluid  Dynamics 

Command,  Control,  Communications,  Computers,  and  Intelligence 

Conceptual  Model  of  the  Mission  Space 

Computer-Numerically-Controlled 

Common  Operating  Digital  Environment 

Capstone  Requirements  Document 

Common  Technical  Framework 

Data  Interface  Format 

Defense  Modeling  and  Simulation  Office 


DoD  Department  of  Defense 
DRA  Decision  Risk  Analysis 
DS  Data  Standards 

DSAC  Defense  Systems  Affordability  Council 
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DSMC  Defense  Systems  Management  College 
DT&E  Developmental  Test  and  Evaluation 
EMD  Engineering  and  Manufacturing  Development 
EXCIMS  Executive  Council  for  Modeling  and  Simulation 
FMTV  Family  of  Tactical  Vehicles 
rOTT  Follow-On  To  TOW 
HIMARS  High  Mobility  Artillery  Rocket  System 
HITL  Human-in-the-Loop 
HLA  High  Level  Architecture 
HWIL  Hardware  in  the  Loop 
ICD  Interface  Control  Document 
IPPD  Integrated  Product  and  Process  Development 
IPX  Integrated  Product  Team 
JIMM  Joint  Interim  Mission  Model 
JMASS  Joint  Modeling  and  Simulation  System 
JROC  Joint  Requirements  Oversight  Council 
JSF  Joint  Strike  Fighter 

JSTARS  Joint  Surveillance  and  Target  Attack  Radar  System 
LCC  Life  Cycle  Cost 
LCM  Life  Cycle  Management 
LFT&E  Live  Fire  Test  and  Evaluation 
LRIP  Low  Rate  Initial  Production 
MAIS  Major  Automated  Information  System 
MDAP  Major  Defense  Acquisition  Program 
MNS  Mission  Need  Statement 
MOE  Measures  of  Effectiveness 
MOO  Measures  of  Outcome 
MOP  Measures  of  Performance 


M&S  Modeling  and  Simulation 
MSIP  Modeling  and  Simulation  Investment  Plan 
MSOSA  Modeling  and  Simulation  Operational  Support  Activity 
MSRR  Modeling  &  Simulation  Resource  Repository 
MTMC  Military  Traffic  Management  Command 
NDIA  National  Defense  Industrial  Association 
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o&s 

ORD 
OSD 
OT&E 
PIM 
PM 
PPBS 
QFD 
ROM 
SBA 
SEDRIS 
SES 
SIPRNET 
SISO 
SMART 
STAF 
STEP  #1 
STEP  #2 
SWIL 
TCO 
TOC 
TOW 
VERT 
W&A 


Operations  and  Support 
Operational  Requirements  Document 
Office  of  the  Secretary  of  Defense 
Operational  Test  and  Evaluation 
Product  Information  Management 
Program  Manager 

Planning,  Programming,  and  Budgeting  System 
Quality  Functional  Deployment 
Rough  Order  of  Magnitude 
Simulation  Based  Acquisition 

Synthetic  Environment  Data  Representation  and  Interchange  Specification 

Simulate,  Emulate,  Stimulate 

Secret  Internet  Protocol  Routing  Network 

Simulation,  Interoperability,  and  Standards  Organization 

Simulation  and  Modeling  for  Acquisition,  Requirements  and  Training 

Simulation  Test  Acceptance  Facility 

Simulation,  Test,  and  Evaluation  Process 

Standard  for  the  Exchange  of  Product  Data 

Software  in  the  Loop 

Total  Cost  of  Ownership 

Total  Ownership  Cost 

Tube-Launched,  Optically-Tracked,  Wire-Guided 
Venture  Evaluation  Review  Technique 
Verification,  Validation,  and  Accreditation 
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GLOSSARY 


Accreditation 

The  official  certification  that  a  model  or  simulation  is  acceptable  for  use  for  a  specific  purpose, 
(reference  DoD  M&S  Glossary) 

Advanced  Concept  Technology  Demonstration 

Technology  demonstrations  that  are  tightly  focused  on  specific  military  concepts  and  that 
provide  the  incorporation  of  technology  that  is  still  at  an  informal  stage  into  a  warfighting 
system.  The  ACTDs  have  three  objectives:  a)  to  have  the  user  gain  an  understanding  of  and 
to  evaluate  the  military  utility  of  concepts  before  committing  to  acquisition;  b)  to  develop 
corresponding  concepts  of  operation  and  doctrine  that  make  best  use  of  the  new  capability; 
and  c)  to  provide  the  residual  operational  capability  to  the  forces.  ACTDs  are  of  militarily 
significant  scope  and  of  a  size  sufficient  to  establish  utility.(reference  DoD  M&S  Glossary) 

Aggregation 

The  ability  to  group  entities  while  preserving  the  effects  of  entity  behavior  and  interaction 
while  grouped,  (reference  DoD  M&S  Glossary) 

Algorithm 

A  prescribed  set  of  well  defined,  unambiguous  rules  or  processes  for  the  solution  of  a  problem 
in  a  finite  number  of  steps,  (reference  DoD  M&S  Glossary) 

Artificial  Intelligence 

The  effort  to  automate  those  human  skills  that  illustrate  our  intelligence  e.g.,  understand¬ 
ing  visual  images,  understanding  speech  and  written  text,  problem  solving  and  medical 
diagnosis,  (reference  DoD  M&S  Glossary) 

Battlespace 

Refers  both  to  the  physical  environment  in  which  the  simulated  warfare  will  take  place  and 
the  forces  that  will  conduct  the  simulated  warfare.  All  elements  that  support  the  front  line 
forces  (e.g.,  logistics,  intelligence)  are  included  in  this  definition  of  battlespace.  (reference 
DoD  M&S  Glossary) 

Benchmark 

The  activity  of  comparing  the  results  of  a  model  or  simulation  with  an  accepted  representa¬ 
tion  of  the  process  being  modeled,  (reference  DoD  M&S  Glossary) 

ffiend  in  the  knee’ 

Refers  to  the  point  where  incremental  increases  in  performance  are  obtained  at  ever 
increasing  costs,  or  the  point  of  diminishing  returns. 
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Common-Use  M&S 

M&S  applications,  services,  or  materials  provided  by  a  DoD  Component  to  two  or  more 
DoD  Components,  (reference  DoD  M&S  Glossary) 

Component 

(See  “DoD  Component”) 

Computational  Fluid  Dynamics 

The  accurate  numerical  solution  of  the  equations  describing  fluid  and  gas  motion  and  the 
related  use  of  digital  computers  in  fluid  dynamics  research,  (reference  DoD  Mission  Success 
from  High  Performance  Computingy  DoD  HPCMO  Report,  Office  of  the  Secretary  of  De¬ 
fense,  DDRE,  March  1995) 

Computer  Simulation 

A  dynamic  representation  of  a  model,  often  involving  some  combination  of  executing  code, 
control/display  interface  hardware,  and  interfaces  to  real-world  equipment,  (reference  DoD 
M&S  Glossary) 

Conceptual  Model  of  the  Mission  Space 

First  abstractions  of  the  real  world  that  serve  as  a  frame  of  reference  for  simulation  devel¬ 
opment  by  capturing  the  basic  information  about  important  entities  involved  in  any  mission 
and  their  key  actions  and  interactions.  They  are  simulation-neutral  views  of  those  entities, 
actions,  and  interactions  occurring  in  the  real  world,  (reference  DoD  M&S  Glossary) 

Constructive  Model  or  Simulation 

Models  and  simulations  that  involve  simulated  people  operating  simulated  systems.  Real 
people  stimulate  (make  inputs)  to  such  simulations,  but  are  not  involved  in  determining  the 
outcomes.  Constructive  simulations  are  often  referred  to  as  war  games.  (Note:  Also  see 
additional  information  under  “Live,  Virtual,  and  Constructive  Simulation.”)  (reference  DoD 
M&S  Glossary) 

Cost  as  an  Independent  Variable 

Methodologies  used  to  acquire  and  operate  affordable  DoD  systems  by  setting  aggressive, 
achievable  lifecycle  cost  objectives,  and  managing  achievement  of  these  objectives  by  trad¬ 
ing  off  performance  and  schedule,  as  necessary.  Cost  objectives  balance  mission  needs  with 
projected  out-year  resources,  taking  into  account  anticipated  process  improvements  in  both 
DoD  and  industry.  CAIV  has  brought  attention  to  the  government’s  responsibilities  for 
setting  and  adjusting  lifecycle  cost  objectives  and  for  evaluating  requirements  in  terms  of 
overall  cost  consequences,  (reference  DSMC  Glossary  of  Defense  Acquisition  Acronyms  and 
Terms,  8th  Edition) 

Decision  Risk  Analysis 

A  methodology  that  quantifies  and  ties  together  cost,  schedule,  performance,  producibility, 
risk,  and  quality  to  permit  tradeoffs. 
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DoD  Component 

One  of  the  four  services  within  the  Department  of  Defense,  i.e.  Army,  Navy,  Air  Force,  or 
Marine  Corps. 

Domain 

The  physical  or  abstract  space  in  which  the  entities  and  processes  operate.  The  domain  can 
be  land,  sea,  air,  space,  undersea,  a  combination  of  any  of  the  above,  or  an  abstract  domain, 
such  as  an  n-dimensional  mathematics  space,  or  economic  or  psychological  domains, 
(reference  DoD  M&S  Glossary) 

Emulate 

To  represent  a  system  by  a  model  that  accepts  the  same  inputs  and  produces  the  same 
outputs  as  the  system  represented.  For  example,  to  emulate  an  8-bit  computer  with  a  32-bit 
computer,  (reference  DoD  M&S  Glossary) 

Emulation 

The  imitation  of  a  computer  system,  performed  by  a  combination  of  hardware  and  soft¬ 
ware,  that  allows  programs  to  run  on  systems  that  would  normally  be  incompatible,  (reference 
http://www.wcom.co  m/cgi-bin/dictQuery.cgi?key=emulation) 

Entity 

A  distinguishable  person,  place,  unit,  thing,  event,  or  concept  about  which  information  is 
kept,  (reference  DoD  M&S  Glossary.) 

Enterprise 

An  arbitrarily-defined  functional  and  administrative  entity  that  exists  to  perform  a  specific, 
integrated  set  of  missions  and  achieve  associated  goals  and  objectives,  encompassing  all  of 
the  primary  functions  necessary  to  perform  those  missions.  An  intranet,  for  example,  is  a 
good  example  of  an  enterprise  computing  system,  (reference  http://www.pcwebopaedia.com/ 
TERM/e/enterprise.html) 

Environment 

The  texture  or  detail  of  the  natural  domain,  that  is  terrain  relief,  weather,  day,  night,  terrain 
cultural  features  (such  as  cities  or  farmland),  sea  states,  etc.;  and  the  external  objects,  con¬ 
ditions,  and  processes  that  influence  the  behavior  of  a  system  (such  as  terrain  relief,  weather, 
day/night,  terrain  cultural  features,  etc.),  (reference  DoD  M&S  Glossary) 

Executive  Council  for  Modeling  and  Simulation 

An  organization  established  by  the  USD(A&T)  and  responsible  for  providing  advice  and 
assistance  on  DoD  M&S  issues.  Membership  is  determined  by  the  USD(A&T)  and  is  at  the 
Senior  Executive  Service,  flag,  and  general  officer  level,  (reference  DoD  M&S  Glossary) 


Federate 

A  member  of  a  High  Level  Architecture  Federation.  All  applications  participating  in  a 
Federation  are  called  Federates.  This  may  include  federation  managers,  data  collectors, 
real  world  (“live”)  systems  (e.g.,  C4I  systems,  instrumented  ranges,  sensors),  simulations, 
passive  viewers  and  other  utilities,  (reference  DoD  M&S  Glossary) 

Federation 

A  named  set  of  interacting  federates,  a  common  federation  object  model,  and  supporting 
Runtime  Infrastructure,  that  are  used  as  a  whole  to  achieve  some  specific  objective,  (reference 
DoD  M&S  Glossary) 

Fidelity 

The  accuracy  of  the  representation  when  compared  to  the  real  world,  (reference  DoD  M&S 
Glossary) 

Hierarchy 

A  ranking  or  ordering  of  abstractions,  (reference  DoD  M&S  Glossary) 

High  Level  Architecture 

Major  functional  elements,  interfaces,  and  design  rules,  pertaining  as  feasible  to  all  DoD 
simulation  applications,  and  providing  a  common  framework  within  which  specific  system 
architectures  can  be  defined,  (reference  DoD  M&S  Glossary) 

Human  Factors 

The  discipline  or  science  of  studying  man-machine  relationships  and  interactions.  The  term 
covers  all  biomedical  and  psychological  considerations;  it  includes,  but  is  not  limited  to, 
principles  and  applications  in  the  areas  of  human  engineering,  personnel  selection,  train¬ 
ing,  life  support,  job  performance  aids,  and  human  performance  evaluation,  (reference  DoD 
M&S  Glossary) 

Human-in-the-Loop 

A  model  that  requires  human  interaction,  (reference  DoD  M&S  Glossary) 

Infrastructure 

An  underlying  base  or  foundation;  the  basic  facilities,  equipment,  and  installations  needed 
for  the  functioning  of  a  system,  (reference  DoD  M&S  Glossary) 

Integrated  Product  and  Process  Development 

An  approach  to  systems  acquisition  that  brings  together  all  of  the  functional  disciplines 
required  to  develop,  design,  test,  produce  and  field  a  system.  This  is  essentially  the  same  as 
Concurrent  Engineering,  (reference  DoD  M&S  Glossary) 
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Integrated  Product  Team 

Integrated  Product  Teams  are  a  means  to  achieve  concurrent  engineering  or  Integrated 
Product  and  Process  Development.  They  are  multi-disciplinary  teams  consisting  of  repre¬ 
sentatives  from  all  disciplines  involved  in  the  system  acquisition  process,  from  requirements 
development  through  disposal.  Having  the  participation  of  all  the  appropriate  disciplines, 
Integrated  Product  Teams  are  often  empowered  to  make  decisions  to  achieve  successful 
development  of  their  particular  product,  (reference  DoD  M&S  Glossary) 

Interoperability 

See:  M&S  Interoperability. 

Life  Cycle  Cost 

The  total  cost  to  the  government  of  acquisition  and  ownership  of  that  system  over  its  useful 
life.  It  includes  the  cost  of  development,  acquisition,  operations  and  support  (to  include 
manpower),  and  where  applicable,  disposal,  (reference  D5MC  Glossary  of  Defense  Acquisition 
Acronyms  and  Terms,  8th  Edition) 

Live  Simulation 

A  simulation  involving  real  people  operating  real  systems.  (Note:  Also  see  additional  infor¬ 
mation  under  “Live,  Virtual,  and  Constructive  Simulation.”)  (reference  DoD  M&5  Glossary) 

Live,  Virtual,  and  Constructive  Simulation 

The  categorization  of  simulation  into  live,  virtual,  and  constructive  is  problematic,  because 
there  is  no  clear  division  between  these  categories.  The  degree  of  human  participation  in 
the  simulation  is  infinitely  variable,  as  is  the  degree  of  equipment  realism.  This  categoriza¬ 
tion  of  simulations  also  suffers  by  excluding  a  category  for  simulated  people  working  real 
equipment  (e.g.,  smart  vehicles).  (Note:  also  see  each  term  separately,  e.g.,  live  simulation) 
(reference  DoD  M&S  Glossary) 

M&S  Infrastructure 

M&S  systems  and  applications,  communications,  networks,  architectures,  standards  and 
protocols,  and  information  resource  repositories,  (reference  DoD  M&S  Glossary) 

M&S  Interoperability 

The  ability  of  a  model  or  simulation  to  provide  services  to  and  accept  services  from  other 
models  and  simulations,  and  to  use  the  services  so  exchanged  to  enable  them  to  operate 
effectively  together,  (reference  DoD  M&S  Glossary) 

Maintainability 

The  ability  of  an  item  to  be  retained  in,  or  restored  to,  a  specified  skill  level,  using  pre¬ 
scribed  procedures  and  resources,  at  each  prescribed  level  of  maintenance  and  repair, 
(reference  DSMC  Glossary  of  Defense  Acquisition  Acronyms  and  Terms,  8th  Edition) 
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Mathematical  Model 

A  symbolic  model  whose  properties  are  expressed  in  mathematical  symbols  and  relation¬ 
ships;  for  example,  a  model  of  a  nation’s  economy  expressed  as  a  set  of  equations.  Contrast 
with:  graphical  model;  narrative  model;  software  model;  tabular  model,  (reference  DoD 
M&S  Glossary) 

Measures  of  Effectiveness  (MOE) 

A  measure  of  operational  success  that  must  be  closely  related  to  the  objective  of  the  mis¬ 
sion  or  operation  being  evaluated.  A  meaningful  MOE  must  be  quantifiable  and  measure 
to  what  degree  the  real  objective  is  achieved,  (reference  DSMC  Glossary  of  Defense 
Acquisition  Acronyms  and  Terms,  8th  Edition) 

Measures  of  Outcome  (MOO) 

Metrics  that  define  how  operational  requirements  contribute  to  end  results  at  higher  levels, 
such  as  campaign  or  national  strategic  outcomes,  (reference  M<&S  Report,  DSMC  1994  Re¬ 
search  Fellows  Report) 

Measures  of  Performance  (MOP) 

Measures  of  the  lowest  level  of  performance  representing  subsets  of  measures  of  effective¬ 
ness  (MOEs).  Examples  are  speed,  payload,  range,  time  on  station,  frequency,  or  other 
distinctly  quantifiable  performance  features,  (reference  DSMC  Glossary  of  Defense 
Acquisition  Acronyms  and  Terms,  S***  Edition) 

Metadata 

Information  describing  the  characteristics  of  data;  data  or  information  about  data;  descrip¬ 
tive  information  about  an  organization’s  data,  data  activities,  systems,  and  holdings, 
(reference  DoD  M&S  Glossary) 

Metric 

A  measure  of  the  extent  or  degree  to  which  a  product  possesses  and  exhibits  a  certain  quality, 
property,  or  attribute,  (reference  DoD  M&S  Glossary) 

Metric(s) 

A  process  or  algorithm  that  may  involve  statistical  sampling,  mathematical  computations, 
and  rule-based  inferencing.  Metrics  provide  the  capability  to  detect  and  report  defects  within 
a  sample,  (reference  DoD  M&S  Glossary) 

Mission  Space 

The  environment  of  entities,  actions,  and  interactions  comprising  the  set  of  interrelated 
processes  used  by  individuals  and/or  organizations  to  accomplish  assigned  tasks,  (reference 
DoD  M&S  Glossary) 
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Model 

A  physical,  mathematical,  or  otherwise  logical  representation  of  a  system,  entity, 
phenomenon,  or  process,  (reference  DoDM&S  Glossary) 

Modeling 

Application  of  a  standard,  rigorous,  structured  methodology  to  create  and  validate  a  physi¬ 
cal,  mathematical,  or  otherwise  logical  representation  of  a  system,  entity,  phenomenon,  or 
process,  (reference  DoD  M&S  Glossary) 

Modeling  &  Simulation  Resource  Repository 

The  MSRR  is  a  collection  of  Modeling  and  Simulation  (M&S)  resources.  MSRR  Resources 
include  models,  simulations,  object  models.  Conceptual  Models  of  the  Mission  Space 
(CMMS),  algorithms,  instance  databases,  data  sets,  data  standardization  and  administra¬ 
tion  products,  documents,  tools  and  utilities,  (reference  www.msrr.dmso.mil/) 

Modeling  and  Simulation 

The  use  of  models,  including  emulators,  prototypes,  simulators,  and  stimulators,  either  stati¬ 
cally  or  over  time,  to  develop  data  as  a  basis  for  making  managerial  or  technical  decisions. 
The  terms  “modeling”  and  “simulation”  are  often  used  interchangeably,  (reference  DoD 
M&S  Glossary) 

Modeling  and  Simulation  Accreditation 

See:  Accreditation. 

Model-Test-Model 

An  integrated  approach  to  using  models  and  simulations  in  support  of  pre-test  analysis  and 
planning;  conducting  the  actual  test  and  collecting  data;  and  post-test  analysis  of  test  results 
along  with  further  validation  of  the  models  using  the  test  data,  (reference  DoD  Glossary) 

Physical  Prototype 

A  model  whose  physical  characteristics  resemble  the  physical  characteristics  of  the  system 
being  modeled;  for  example,  a  plastic  or  wooden  replica  of  an  airplane.  A  mock-up.  (reference 
DoD  M&S  Glossary) 
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Planning,  Programming,  and  Budgeting  System  (PPBS) 

The  primary  resource  allocation  process  of  DoD.  One  of  three  major  decision  making  support 
systems  for  defense  acquisition  (the  other  two  are  the  Requirements  Generation  System 
and  the  Acquisition  Management  System).  It  is  a  formal,  systematic  structure  for  making 
decisions  on  policy,  strategy,  and  the  development  of  forces  and  capabilities  to  accomplish 
anticipated  missions.  PPBS  is  a  cyclic  process  containing  three  distinct,  but  interrelated 
phases:  planning,  which  produces  Defense  Planning  Guidance  (DPG);  programming,  which 
produces  approved  program  objectives  memorandum  (POM)  for  the  military  departments 
and  defense  agencies;  and  budgeting,  which  produces  the  DoD  portion  of  the  President’s 
national  budget,  (reference  DSMC  Glossary  of  Defense  Acquisition  Acronyms  and  Terms,  8^*" 
Edition) 

Producibilify 

The  relative  ease  of  manufacturing  an  item  or  system.  This  relative  eeise  is  governed  by  the 
characteristics  and  features  of  a  design  that  enables  economical  fabrication,  assembly, 
inspection  and  testing  using  available  manufacturing  techniques,  (reference  D5MC  Glossary 
of  Defense  Acquisition  Acronyms  and  Terms,  8th  Edition) 

Reliability 

The  ability  of  a  system  and  its  parts  to  perform  its  mission  without  failure,  degradation,  or 
demand  on  the  support  system,  (reference  D5MC  Glossary  of  Defense  Acquisition  Acronyms 
and  Terms,  8*^^  Edition) 

Reliability  Model 

A  model  used  to  estimate,  measure,  or  predict  the  reliability  of  a  system;  for  example,  a 
model  of  a  computer  system,  used  to  estimate  the  total  down  time  that  will  be  experienced, 
(reference  DoD  M&S  Glossary) 

Risk 

Risk  is  a  measure  of  the  inability  to  achieve  a  program’s  defined  performance,  schedule, 
and  cost  objectives,  and  has  two  components:  1)  The  probability  of  failing  to  achieve  par¬ 
ticular  performance,  schedule,  or  cost  objectives;  and  2)  The  consequence  of  failing  to  achieve 
those  objectives,  (reference  DSMC  Acquisition  Strategy  Guide,  Third  Edition) 

SBAPull 

Those  forces  (attitudes,  expectations,  incentives,  directives,  policies,  etc.),  external  to  a 
program  office,  that  influence  a  program  office  to  adopt  an  SBA  approach. 
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SBA  Push 

Those  forces  (attitudes,  expectations,  incentives,  directives,  policies,  etc.),  internal  to  a 
program  office,  that  influence  a  program  office  to  adopt  an  SBA  approach. 

Simulation  Based  Acquisition 

An  iterative,  integrated  product  and  process  approach  to  acquisition,  using  modeling  and 
simulation,  that  enables  the  warfighting,  resource  allocation,  and  acquisition  communities 
to  fulfill  the  warfighter’s  materiel  needs,  while  maintaining  Cost  As  an  Independent  Variable 
(CAIV)  over  the  system’s  entire  lifecycle  and  within  the  DoD’s  system  of  systems. 

Solid  Modeling 

A  digital  representation  of  the  surface  characteristics  of  an  object. 

Stimulate 

To  provide  input  to  a  system  in  order  to  observe  or  evaluate  the  system’s  response,  (reference 
DoD  M&S  Glossary) 

Stimulation 

The  use  of  simulations  to  provide  an  external  stimulus  to  a  system  or  subsystem.  An  example 
is  the  use  of  a  simulation  representing  the  radar  return  from  a  target  to  drive  (stimulate)  the 
radar  of  a  missile  system  within  a  hardware/software-in-the-loop  simulation,  (reference  DoZ) 
M&S  Glossary) 

Supportability 

The  degree  of  ease  to  which  system  design  characteristics  and  planned  logistics  for  resources, 
including  the  logistic  support  elements,  allows  for  the  meeting  of  system  availability  and 
wartime  utilization  requirements,  (reference  Glossary  of  Defense  Acquisition  Acronyms 

and  Terms,  8th  Edition) 

Synthetic  Environment 

Simulations  that  represent  activities  at  a  high  level  of  realism,  from  simulations  of  theaters 
of  war  to  factories  and  manufacturing  processes.  These  environments  may  be  created  within 
a  single  computer  or  a  vast  distributed  network  connected  by  local  and  wide  area  networks 
and  augmented  by  super-realistic  special  effects  and  accurate  behavioral  models.  They  allow 
visualization  of  and  immersion  into  the  environment  being  simulated,  (reference 
Glossary) 
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System  of  Systems 

Seamless  connectivity  of  Command,  Control,  Communications,  Computers,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  across  the  sea-land-space  interface  in  a  joint 
warfighting  environment,  and  assimilation.., into  a  coherent  tactical  picture... to  develop  a 
multi-dimensional  netted  architecture... while  providing  rapid  sensor- to-shooter  connec¬ 
tivity.  (reference  Admiral  Jay  Johnson,  Chief  of  Naval  Operations,  during  a  visit  to  the 
Joint  Strike  Fighter  program,  1995).  The  authors  use  the  term  System  of  Systems  in  a  con¬ 
text  broader  than  C4ISR,  to  encompass  interactions  between  sub-systems  of  the  same  sys¬ 
tem,  multiple  system  integration,  user  tradeoff  analysis  and  optimization,  and  warfighting 
scenario  optimization.  One  of  the  first  places  for  the  term  System  of  Systems  to  show  up  in 
print  was  in  the  U.S.  Naval  Institute  Proceedings  May  1995  article  “The  Emerging  System 
of  Systems,”  by  Admiral  William  A.  Owens,  US  Navy.  To  the  authors’  knowledge,  there  is 
no  commonly  accepted  definition  of  this  term  as  yet. 

Total  Cost  of  Ownership 

See:  Total  Ownership  Cost  (TOC). 

Total  Ownership  Cost 

The  sum  of  all  financial  resources  necessary  to  organize,  equip,  and  sustain  military  forces 
sufficient  to  meet  national  goals  in  compliance  with  all  laws;  all  policies  applicable  to  DoD; 
all  standards  in  effect  for  readiness,  safety,  and  quality  of  life;  and  all  other  official  mea¬ 
sures  of  performance  for  DoD  and  its  components,  (reference  1998  Annual  Report  to  the 
President  and  the  Congress,  Office  of  the  Secretary  of  Defense) 

Validation 

The  process  of  determining  the  degree  to  which  a  model  or  simulation  is  an  accurate 
representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model  or 
simulation,  (reference  DoD  M&S  Glossary) 

Verification 

The  process  of  determining  that  a  model  or  simulation  implementation  accurately  represents 
the  developer’s  conceptual  description  and  specification.  Verification  also  evaluates  the 
extent  to  which  the  model  or  simulation  has  been  developed  using  sound  and  established 
software-engineering  techniques,  (reference  DoD  M&S  Glossary) 

Verification,  Validation,  and  Accreditation 

See  each  separate  term. 

Virtual 

The  essence  or  effect  of  something,  not  the  fact,  (reference  DoD  M&S  Glossary) 
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Virtual  Battlespace 

The  illusion  resulting  from  simulating  the  actual  battlespace.  (reference Glossary) 

Virtual  Manufacturing 

A  model  or  simulation  of  the  processes  required  for  making  an  item. 

Virtual  Prototype 

A  model  or  simulation  of  a  system  placed  in  a  synthetic  environment,  and  used  to  investi¬ 
gate  and  evaluate  requirements,  concepts,  system  design,  testing,  production,  and  sustainment 
of  the  system  throughout  its  lifecycle,  (reference  DoD  M&S  Glossary) 

Virtual  Reality 

The  effect  created  by  generating  an  environment  that  does  not  exist  in  the  real  world.  Usually, 
virtual  reality  is  a  stereoscopic  display  and  computer-generated  three-dimensional 
environment  which  has  the  effect  of  immersing  the  user  in  that  environment.  This  is  called 
the  immersion  effect.  The  environment  is  interactive,  allowing  the  participant  to  look  and 
navigate  about  the  environment,  enhancing  the  immersion  effect,  (reference  DoD  M&S 
Glossary) 

Virtual  Simulation 

A  simulation  involving  real  people  operating  simulated  systems.  Virtual  simulations  inject 
human-in-the-loop  in  a  central  role  by  exercising  motor  control  skills  (e.g.,  flying  an  air¬ 
plane),  decision  skills  (e.g.,  committing  fire  control  resources  to  action),  or  communication 
skills  (e.g.,  as  members  of  a  C4I  team).  (Note:  Also  see  additional  information  under  “Live, 
Virtual,  and  Constructive  Simulation.”)  (reference  DoD  M&S  Glossary) 

Virtual  World 

See  synthetic  environment. 

Visualization 

The  formation  of  an  artificial  image  that  cannot  be  seen  otherwise.  Typically,  abstract  data 
that  would  normally  appear  as  text  and  numbers  is  graphically  displayed  as  an  image.  The 
image  can  be  animated  to  display  time  varying  data,  (reference  DoD  M&S  Glossary) 

War  Game 

A  simulation  game  in  which  participants  seek  to  achieve  a  specified  military  objective  given 
pre-established  resources  and  constraints;  for  example,  a  simulation  in  which  participants 
make  battlefield  decisions  and  a  computer  determines  the  results  of  those  decisions.  Also 
used  synonymously  with  constructive  simulation,  (reference  DoD  M&S  Glossary) 
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Wargaming 

The  act  of  conducting  a  war  game. 


What-if  Analysis 

Analysis  conducted  to  determine  the  impact  of  changing  a  variable  during  the  design  of  a 
weapon  system,  as  in  “What  if  we  change  the  stiffness  of  the  beam,  what  will  be  the  impact 
on  the  cost,  schedule,  and  performance  of  the  system?” 
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APPENDIX  B 

INTERVIEWS  AND  PERSONAL  CONTACTS 


*  SBA  Industry  Steering  Group  Member  ’ 
**  Joint  SBA  Task  Force  Member^ 


COMPANY 

CONTACT 

ADVANCED  AMPfflBIOUS  ASSAULT 
VEHICLE  (AAAV)  PROGRAM  OFFICE 

AAAV  Technology  Center 

USMC  991  Annapolis  Way 

Woodbridge,  VA  22191-1215 

Richard  Bayard 

Assistant  Program  Manager 

rbayard@notes.hqi.usmc.mii 

703-492-3344 

Mark  Routson 

General  Dynamics 
routsonm@gdls.com 

703-492-3222 

AERONAUTICAL  SYSTEMS  CENTER  (ASC) 
Distributed  Mission  Training  Office 

Wright  Patterson  Air  Force  Base,  OH 

Dr.  Jerry  Arnett 
arnettjb@asc-yw.wpafb.af.mil 

AIR  FORCE  MATERIAL  COMMAND 

Modeling  and  Simulation  Integration  Office 
Operating  Location 

12350  Research  Parkway 

Orlando,  FL  32826 

Lt  Col  Joseph  M.  Terry** 
AFMC/MSIO-OL  Liaison  Officer 
terry.joe@afams.af.mil 

407-208-5762 

AIR  FORCE  RESEARCH  LABORATORY 

6001  S.  Power  Road,  Bldg.  558 

Mesa,AZ  85206-0904 

Col  Lynn  Carroll,  Chief 

Warfighter  Training  Research  Division 

ANSER 

1215  Jefferson  Davis  Highway 

Arlington,  VA  22202 

Dr.  Aron  Pinker 
pinkera@anser.org 

703-416-3439 

APPLIED  SYSTEMS  INTELLIGENCE  INC. 
10882  Crabapple  Road,  Suite  2 

Roswell,  Georgia  30075 

Norm  Geddes 

President 

norm@asinc.com 
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COMPANY 


CONTACT 


ARMY  SOFTWARE  REUSE  CENTER 
ASQB-IRC  (Stop  C-2) 

6000  e'"  St,  Suite  S122A 

Ft  Belvoir,  VA  22060-5576 

LTC  Gene  Glasser 

Director,  ASRC 

glassere@issc.belvoir.army.mil 

806-3860/4300 

ASSISTANT  SECRETARY  OF  THE  ARMY 
(RESEARCH,  DEVELOPMENT  AND 
ACQUISITION),  ASA(RDA) 

Attn:  SARD-DO 

103  Army  Pentagon 

Washington,  D.C.  20310 

Lieutenant  General  Paul  J.  Kern 

Military  Deputy  to  the  ASA(RDA) 
703-697-0356 

Ellen  Purdy** 

Senior  Operations  Research  Analyst 
purdye@sarda.army.mil 

703-604-7006 

COL  Michael  Lavine 
lavinem@sarda.army.mil 

703-604-7111 

Dr.  Herbert  Fallin 

Director  for  Assessment  and  Evaluation 
fallinh@sarda.army.mil 

703-697-2653 

Mike  Truelove** 

Acquisition  Analyst 

Science  Applications  International 
Corporation  (SAIC) 
truelovm@sarda.army.mil 

703-604-7013 

Paul  Amos 

Acquisition  Analyst 

Science  Applications  International 
Corporation  (SAIC) 
amosp@sarda.army.mil 

703-604-7003 

ATACMS-BAT  PROJECT  OFFICE 
ArmyTACMS-BAT 

ATTN:  SFAE-MSL-AB 

Redstone  Arsenal,  AL  35898-5650 

COL  John  Holly 
hollyj  @redstone,army.mil 

205-876-1141 

COMPANY 


CONTACT 


BALL  AEROSPACE 

8381  Old  Courthouse  Road 

Vienna,  VA  22182 

Larry  Jobson 
ljobson@balLcom 

703-917-9125 

BARRONCAST,  INC. 

215  W.  Oakwood  Rd. 

P.O.  Box  138 

Oxford,  Michigan  48371 

Bruce  Barron 

Vice  President  &  General  Manager 
Bruce@barroncast.com 

248-628-4300 

Lee  Bennett 

Controller 

lee@barroncast.com 

248-628-4300 

Merlin  Warner 

Sales  Engineer 

Sales@barroncast.com 

BOEING  COMPANY 

Seattle,  WA  98124 

Larry  Winslow.  Director 

Technology  for  Phantom  Works 

Michael  W  Johnson* 

Defense  &  Space  Group 

Vice-President  Modeling  &  Simulation 

michael.w.johnson@boeing.com 

253-773-3055 

CACI 

1600  Wilson  Blvd 

Arlington,  VA  22209-2515 

Dr.  Klaus  Dannenberg 

Senior  Vice  President 
kdannenberg@hq.caci.com 

703-558-0255 

CARDEROCK  NAVAL  SURFACE 
WARFARE  CENTER 

9500  MacArthur  Blvd  West 

Bethesda,  MD  20817-5700 

Myles  Hurwitz 

Head,  Computer  M&S  Department 
NSWC/Carderock 
mhurwitz@oasys.dt.navy.mil 
(301)227-1927 

CENTER  FOR  NAVAL  ANALYSIS 

4400  Ford  Avenue 

Alexandria,  VA  22302 

Dr.  Henry  Eskew 

703-824-2254 
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COMPANY 


CONTACT 


CHRYSLER  CORPORATION 

800  Chrysler  Drive 

Auburn  Hills,  MI  48326-2757 

Arthur  Anderson 

Manager,  Vehicle  Engineering  & 
Dimension  Control 

Large  Car  Platform 

Greg  Avesian 

Manager,  Solutions  Marketing 
gjal@chiysler.com 

248-576-6676 

CLOSE  COMBAT  ANTI-TANK  WEAPON 
SYSTEM  (CCAWS)  PROJECT  OFFICE 
SFAE-MSL-CC-TM 

Redstone  Arsenal,  AL  35898 

John  Bier 

Technical  Manager 

Follo^v  on  to  TOW  Program 

john.bier@msl2.redstone.army.mil 

256-842-0494 

Allen  Zumbach 

256-876-3200 

COLEMAN  RESEARCH  CORPORATION 

6820  Moquin  Drive 

Huntsville,  AL  35806 

Tim  Thornton 
tthomton@hsv.crc.com 

205-922-6000 

COMANCHE  PROGRAM  OFFICE 

Redstone  Arsenal,  AL  35898 

Dan  Hollenbaugh 

Program  Test  Engineer 

256-313-4464 

COMPUTER  SCIENCES  CORPORATION 
China  Lake,  CA 

Dennis  Laack 

Joint  Accreditation  Support  Activity 
(JASA) 

dlaack@csc.com 

805-987-9641  xl30 
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COMPANY 


CONTACT 


CRUSADER  PROGRAM  OFFICE 
SFAE-GCSS-CRE,  Building  171 

Picatinny  Arsenal,  NJ  07806-5000 

Col  Bill  Sheaves 

Program  Manager 
wsheaves@pica.army.mil 

973-724-4588 

Kevin  Fahey 

Deputy  Program  Manager 
kfahey@pica.army.mil 

973-724-4588 

Jim  Shields 

Chief  of  Systems  Engineering 
jshields@pica.army.mil 

973-724-7608 

DEFENSE  ADVANCED  RESEARCH 
PROJECTS  AGENCY  (DARPA) 

5201  Leesburg  Rke,  Suite  503 

Falls  Church,  VA  22041 

Gary  Jones,  Director 

Advanced  Technology  Integration 
gjones@darpa.mil 

703-681-1484 

DEFENSE  INTELLIGENCE  AGENCY 
Bolling  Air  Force  Base 

Washington,  D.C,  20340-5100 

Richard  Bernstein** 

Intelligence  Officer 
bemstr@acq.osd.mil 

703-820-1623 

B-7 


COMPANY 


CONTACT 


DEFENSE  MODELING  AND 

SIMULATION  OFFICE 

1901  N.  Beauregard  St 

Alexandria,  VA  22311 

Bruce  Bailey 

DoD  M&S  Working  Group 
bailey@dmso.mil 

703-824-3420 

Col  Kenneth  “Crash”  Konwin 
konwin@dmso.mil 

703-998-0660 

Dr.  Judith  Dahmann 
jdahmann@msis.dmso.mil 

703-998-0660 

Jona  McKee 

jmckee@msosa.dmso.mil 

703-998-1637 

John  Gray 

Senior  M&S  Consultant 
jgray@msosa.dmso.mil 

703-998-1623 

Heikki  Joonsar** 
hjoonsar@msis.dmso.mil 

703-575-2452 

DEFENSE  SYSTEMS  MANAGEMENT 
COLLEGE 

9820  Belvoir  Rd 

Fort  Belvoir,  VA  22060 

Shane  Donahue 

donahue_patrick@dsmc.dsm.mil 

703-805-5411 

Randy  Zittel 

Professor  Systems  Engineering 
Zittelr@dsmc.dsm.mil 

703-805-5267 

DEPUTY  CHIEF  OF  NAVAL  OPERATIONS 
(LOGISTICS),  OPNAV  N4 

Navy  Pentagon  2000 

Washington,  DC  20350 

Dan  Fink,  Deputy 

Acquisition  Logistics  Integration 
fink.dan@hq.navy.mil 

703-601-1679 

COMPANY 


CONTACT 


DEPUTY  UNDERSECRETARY  FOR 
DEFENSE,  (INDUSTRIAL  AFFAIRS  & 
INSTALLATIONS),  INDUSTRIAL 
CAPABILITIES  AND  ASSESSMENT 

5203  Leesburg  Pike,  Suite  1403 

Falls  Church,  VA  22041-3466 

John  D.  Binford** 

General  Engineer 
jbinford@acq.osd.mil 

703-681-5472 

DRAPER  LABORATORY,  INC. 

The  Charles  Stark 

555  Technology  Square 

Cambridge,  MA  02139-3563 

Dr  James  Negro 
jnegro@draper.com 

617-258-3860 

Moshe  Cohen 
moshe@draper.com 

617-258-4425 

ELECTRIC  BOAT  CORP 

75  Eastern  Point  Road 

Groton,  CN  06340-4989 

Gregory  Angelini* 

Software  Technology  Team  Leader, 
Computer  Systems  Technology 

John  Porter 

CVX  Principal  Engr. 
jporter@ebmail.gdeb.com 

860-433-7342 

Jim  Boudreaux 
jboudrea@ebmail.gdeb.com 

860-433-6378 

ELECTRONIC  SYSTEMS  COMMAND 

5  Eglin  St 

Hanscom  AFB,  MA  01731-2100 

COL  Hoot  Gibson 

Director,  Modeling,  Simulation  and 
Training  PAD 

gibsonh@wg.hanscom.af.mil 

781-377-6554 

EVANS  AND  SUTHERLAND 

12443  Research  Parkway,  Suite  303 

Orlando,  FL  32826 

Paul  Hinote 
phinote@es.com 

407-658-1802 
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FOCUS:  HOPE 

1400  Oakman  Blvd 

Detroit,  MI  48238 

Joe  Petrosky 

General  Manager 

Center  for  Advanced  Technologies 

petrosky@exchangel.focushope.edu 

www.focushope.edu 

313-494-4274 

FORD  MOTOR  COMPANY 

24500  Glendale  Ave 

Detroit,  MI  48239 

Gene  Coffman 

Staff  Technical  Specialist 
gcoffman@ford.com 

313-592-2079 

Hwa-Sung  Na 

Principal  Engineering  Specialist 
hna@ford.com 

313-592-2707 

FORD  MOTOR  COMPANY 

2000  Rotunda  Drive 

Dearborn,  MI  48121-2053 

Michael  Carty 

Product  Development  Center 
Mcarty@ford.com 

313-317-9213 

Richard  Riff 

Advanced  Engineering  Center  Manager 
C3P  Project  Office 

Rriff@ford.com 

313-337-9670 

GENERAL  MOTORS 

30200  Mound  Road 

Warren,  MI  48090-9010 

David  Chang,  Director 

National  Automotive  Organization 
Vehicle  Synthesis,  Analysis  & 

Simulation  Process  Center 
LNUSTCl.FZ9MR7@gmeds.com 
810-986-6880 

Prakash  Shrivastava 

LNUSTCl.LZlNYJ@gmeds.com 

810-986-2206 


COMPANY 

CONTACT 

HARVARD  BUSINESS  SCHOOL 

Soldiers  Field  Road 

Morgan  Hall  Boston,  MA  0216 

Professor  Dorothy  Leonard 

617-495-6637 

Stefan  Thomke 

Assistant  Professor 
sthomke@hbs.harvard.edu 

617-495-6569 

Ron  Fox 

Professor  Emeritus 
rfox@hbs.edu 

315-655-4694 

HEADQUARTERS  DEFENSE 
INTELLIGENCE  AGENCY 

Bolling  AFB,  MD 

Richard  Bernstein 
rbernste@MSIS.dmso.mil 

202-231-8934 

HIGH  MOBILITY  ARTILLERY 

ROCKET  SYSTEM  (HIMARS) 

1411  Westview  Drive 

Coralville,  lA  52241 

Mr.  Brock  Birdsong 

Mechanical  Engineer,  Missile  RDEC 

birdsong-cb@redstone.army.mil 

319-339-8755 

IDA 

1801  North  Beauregard  Street 

Alexandria,  VA  22311-1772 

Hans  Mair 

Operational  Evaluation  Division 
mairh@asme.org 

703-845-2034 

INSTITUTE  FOR  SIMULATION 
TECHNOLOGY 

3280  Progress  Drive 

Orlando,  FL  32826 

Dr.  Ronald  Hofer,  Ph.D. 

Associate  Director 
rhofer@ist.ucf.edu 

407-658-5576 

Ernie  Smart 
Director 

Governmental  &  Industrial  Partner¬ 
ships 

esmart@ist.ucf.edu 

407-658-5014 
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INTEGRATION  &  TECHNOLOGY 

BATTLE  LAB 

CDR,  USATRADOC 

Attn:  ATCD-B 

Ft  Monroe,  VA  23651-5000 

Bob  Dodd 

Military  Analyst 
doddr@monroe.army.mil 

757-727-5715 

J.T.  PARKER  ASSOCIATES 

3061  Mimon  Road 

Annapolis,  MD  21403-1317 

Ted  Parker* 

President;  ADM  (Ret) 
jparker@tecnetl.jcte.jcs.mil 

301-261-1216 

JOHN  J.  MCMULLEN  ASSOCIATES,  INC 
2341  Jefferson  Davis  Hwy 

Arlington,  VA  22202 

Rob  Beadling* 

Program  Manager 

CVX  Advanced  Simulation 
rbeadling@jjma.com 

703-418-4273 

JOHNS  HOPKINS  UNIVERSITY 

801  University  Boulevard,  SE 

Albuquerque,  NM  87106 

Dennis  Lester 
dennis.lester@jhuapl.edu 

505-853-1975 

JOHNS  HOPKINS  UNIVERSITY 

Applied  Physics  Laboratory 

11100  Johns  Hopkins  Road 

Laurel,  MD  20723-6099 

Dr.  Mark  Moulding 

Physicist 

mark.moulding@jhuapl.edu 

240-228-4979 

Glenn  Gealy 

gienn.gealy@jhuapl.edu  2 

40-228-6855 

Jim  Coolahan** 

SBA  Task  Force  Technical  Support 
James.coolahan@jhuapl.edu 

240-228-5155 

Bob  Lutz** 

SBA  Task  Force  Technical  Support 
Robert.lutz@jhuapl.edu 

240-953-5000 

B-12 


COMPANY 

CONTACT 

JOINT  SIMULATION  SYSTEM  (JSIMS) 
Orlando,  FL 

Dave  Walker 

Analysis  and  Integration  Manager 
david_walker@jsims.com 

407-384-2909,  ext.  738 

JOINT  STRIKE  FIGHTER 

PROGRAM  OFFICE 

1745  Jeff  Davis  Hwy,  Suite  307 

Arlington,  VA  22202 

COL  Phil  Faye 

Director,  Requirements  (RQ) 
fayepa@jast.mil 

703-602-7390x6649 

KROUSE  ASSOCIATES 

7310  Holly  Park  Dr. 

Concord,  OH  44060 

John  Krouse 

jkrouse@compuserve.com 

440-354-5334 

LOCKHEED  MARTIN 

Washington  Operations 

1725  Jefferson  Davis  Highway 

Arlington,  VA  22202-4127 

A.J.  (Beau)  Beauregard,  P.E.* 

Director,  Advanced  Programs 
Aeronautics  Sector 
beau.beauregard@lmco.com 
703-413-5711 

LOCKHEED  MARTIN 

PO  Box  748 

Fort  Worth,  TX  76101 

Linda  Poole 

Program  Manager 

Virtual  Product  Development  Initiative 
(VPDI) 

817-763-2096 

LOCKHEED  MARTIN  AERONAUTICAL 
SYSTEMS 

86  South  Cobb  Drive 

Marietta,  GA  30063 

Margaret  Herald* 

margaret.s.herald@lraco.com 

770-494-9833 

LORAIVLOCKHEED  MARTIN 

FEDERAL  SYSTEMS 

546  Lob  Lolly  Dr 

Woodlake,  NC 

LTG  (Ret)  Dan  Schroeder 

Consultant 

Frigatel20@aol.com 

910-245-3136 

MICROSOFT 

One  Microsoft  Corporation 

Redmond,  WA  98052-6399 

Mark  Walker 
markwalk@microsoft.com 

425-703-3181 
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MITRE 

11493  Sunset  Hills  Rd 

Reston,  VA  20190 

Marvin  Hammond 
mhammond@mitre.org 

703-883-5867 

Stuart  Starr 
starr@mitre.org 

703-883-5494 

MULTIGEN,  INC. 

6701  Democracy  Blvd 

Bethesda,  MD  20817 

Bill  MacKrell 

N.E.Regional  Sales  Representative 
bmackrell@multigen.com 

301-571-2466 

MULTIGEN,  INC. 

550  S,  Winchester  Blvd 

San  Jose,  CA  95128 

Ray  Homan 

Exec.  V.  President  and  Gen.  Manager 
rhoman@multigen.com 

408-261-4100 

NAVAL  COASTAL  SYSTEMS  STATION 

Panama  City,  FL 

Roger  Leete 

leete  roger@ccmail.ncsc.navy.mil 
850-234-4265 

NAVAL  RESEARCH  LABORATORY 

4555  Overlook  Avenue,  SW 

Washington,  D.C.  20375 

Bill  Sandberg 

Deputy  Director,  Laboratory  for 
Computational  Physics  and 

Fluid  Dynamics 
sandberg@lcp.nrl.navy.mil 

202-767-0526 

NAVAL  SEA  SYSTEMS  COMMAND  (NAVSEA) 
2541  Jefferson  Davis  Hwy 

Alexandria,  VA  22242-5610 

Mike  Rice 

03KQ 

rice_mike_l@hq.navsea.navy.mil 
703-602-0887,  ext.  327 

Stuart  Marcus** 

05K11 

marcus_stuartj@hq.navsea.navy.mil 
703-602-7242,  ext.  107 
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NAVAL  SURFACE  WARFARE  CENTER 

Port  Hueneme  Division 

Building  1380 

4363  Missile  Way 

Port  Hueneme,  CA  93043-4307 

Joe  Stelling** 

SBA  Task  Force  Leader 

805-228-8719 

NAVAL  UNDERSEA  WARFARE  CENTER 

1176  Howell  St 

Newport,  RI 02841 

Jeff  Feaster 

Engineering  ,T&E  Dept 

feasterjt@npri70.npt.nuwc.navy.mil 

401-841-6892x22314 

Debra  Jones 

Electronics  Engineer 

Capital  Purchase  Program  Manager 
401-841-2012  X  23351 

Tom  Kowalczyk 

kowalczyktw@code80.npt.nuwc.navy.mil 

401-841-6892 

NAVY  M&S  MANAGEMENT  OFFICE 

Crystal  City  Presidential  Towers 

Arlington,  VA  22202 

CAPT  Jay  Kistler 
kistler.jay@hq.navy.mil 

703  601-1482 

NAVY  MODELING  AND  SIMULATION 
MANAGEMENT  OFFICE 

N6M1 

George  Phillips 

Navy  M&S  Point  of  Contact 
phillips.george@hq.navy.mil 
703-685-6995 

NEWPORT  NEWS  SHIPBUILDING 

4101  Washington  Ave 

Newport  News,  VA  23607-2770 

Wayne  Evans 

Manager,  Lifecycle  Engineering 
evans_wa@nns.com 

757-380-4876 

Dan  Selfridge 
Lifecycle  Engineering 
self ridgedj  ©nns.com 
757-688-6335 
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NORTHROP  GRUMMAN 

8900  East  Washington  Blvd 

Pico  Rivera,  CA  90660 

Rahim  Munshi 

Systems  Engineer 

rahim_munshi@emaiLnorthgrum.com 

562-948-8824 

NSSC,  SC-21  PROGRAM 

2531  Jefferson  Davis  Hwy 

Arlington,  VA  22242 

Janet  Jaensch 

Director,  SC-21  Modeling  &  Simulation 
jaenschjanet@hq.navsea.navy.mil 
703-602-6453,  X  105 

Thien  Ngo 

Asst.  Director,  Modeling  and  Simulation 
ngoJhien_c@hq.navsea.navy.mil 
703-602-6453,  x  122 

OCI,  INCORPORATED 

1235  Jefferson  Davis  Highway 

Arlington,  VA  22202 

Donald  Hirsch 
dhirsch@oc-inc.com 

703-416-0002 

OPTIMETRICS,  INC.  (OMI) 

1  Newport  Drive 

Forest  Hill,  MD  21050 

Frank  Wysoki* 
wysoki@omi.com 

703-791-2286 

ORIGINAL  SIM 

5524  St.  Patrick 

Montreal,  Canada  H4E  1A8 

Christian  Rochefort 

Regional  Sales  Manager 
crochefort@originalsim.com 

514-766-8868 

PLYNETICS  EXPRESS 

1067  Centre  Road 

Auburn  Hills,  MI  48326 

Rick  Hannan 

Project  Manager 

Rhannan@plynex.com 

248-373-4400 

Steven  Willis 

Program  Manager 
swillis@plynex.com 
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PONTIAC-GENERAL  MOTORS  CORE 

100  Renaissance  Center 

Detroit,  MI  48243-7301 

Ronald  Strayhom 

Vice  President,  Sales  and  Service 
rstrayhom@pmd72.hbs.edu 

313-667-4132 

PROGRAM  EXECUTIVE  OFFICER 

THEATER  AIR  DEFENSE  (TAD) 
PEOTAD.NAVSEA 

Crystal  City 

Arlington,  VA 

Lorraine  Shea 

Director  M&S,  PEG  TAD,  NAVSEA 
Shea_Lorraine@hq.navsea.navy.mil 

RAYTHEON  ELECTRONIC  SYSTEMS 

1001  Boston  Post  Rd 

Marlboro,  MA  01752 

William  Stewich 

Mgr.  Business  Dev.  C4I  Programs 
william_stewich@ccmail.res.ray.com 

508-490-3228 

RAYTHEON  SYSTEMS  COMPANY 

Plano,  TX  75086 

Randy  Boys* 
rboys@rtis.ray.com 

972-575-6128 

RAYTHEON  SYSTEMS  COMPANY 

7700  Arlington  Blvd 

Falls  Church,  VA  22204 

Steve  Olson* 

steve_olson@fallschurch.esys.com 

703-876-1942 

RL  ENGWALL  &  ASSOCIATES 

560  Choptank  Cove  Court 

Annapolis,  MD  21401 

Dick  Engwall* 

Management  Consultant 
rlengwall@aol.com 

410-571-8623 

SCIENCE  APPLICATIONS  INTERNATIONAL 
CORPORATION  (SAIC) 

8301  Greensboro  Dr.,  Suite  290 

McLean,  VA  22102 

Annie  Patenaude 

anne.m.patenaude@cpmx.saic.com 

703-749-5109 
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SCIENCE  APPLICATIONS  INTERNATIONAL 
CORPORATION  (SAIC) 

2511  Jefferson  Davis  Highway 

Arlington,  VA  22202-3911 

Tom  Brown 

f.thomas.brown@cpmx.saic.com 

Dave  Thomen 
thomend@mail.etas.com 

703-414-0189 

SCIENCE  APPLICATIONS  INTERNATIONAL 
CORPORATION  (SAIC) 

3045  Technology  Parkway 

Orlando,  FL  32826-3299 

Col  (Ret)  James  Shiflett 

Vice  President 

407-207-2725 

W.  San  Horton 

Senior  Technical  Manager 

407-282-6700 

SECRETARY  OF  THE  AIR  FORCE 

1060  Air  Force  Pentagon 

Washington,  DC  20301 

Lieutenant  General  George  Muellner 
Principal  Deputy  (Acquisition) 
703-695-7311 

MAJ  Gary  Hopper 

SAF/AQPS 

hopperg@af.pentagon.mil 

703-588-7171 

MAJ  Gerald  S.  Smither,  Jr.** 

SAF/AQIK 

smitherg@af.pentagon.mil 

703-588-7191 

Lt.  Col  Paul  W.  Coutee** 
SAF/AQRE 

couteep@af.pentagon.mil 

703-588-5757 
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SIMULATION  TRAINING  AND 
INSTRUMENTATION  COMMAND 
(STRICOM) 

12350  Research  Parkway 
Orlando,  FL  32826 


SPACE  &  WARFARE  SYSTEMS 
COMMAND  (SPAWAR) 

4301  Pacific  Hwy  C60  Topside 
San  Diego,  CA  92110-3127 


Stan  Goodman 
goodmans@stricom.army.mil 

Donald  Jones 
Deputy  Product  Manager, 

Air  &  Combat  Training  Systems 

jonesl@stricom.army.mil 

407-384-5142 

Les  Curless 

les_curless@stricom.army.mil 

407-384-3860 

Dr.  Stuart  Olson 

stuart_olson@stricom.army,mil 

407-384-5132 

Admiral  Piper 

pipera@stricom.army.mil 

407-384-3935 

Phillip  Sprinkle 

Deputy  PM  Training  Devices 

sprinklp@stricom.army.mil 

407-384-5202 

Ed  Trier 

triere@stricom.army.mil 

407-384-3800 

Bill  Blanding 

blandinb@stricom.army.mil 

407-384-5118 

Phillip  Homick 

Deputy  Program  Manager  M&S 

homickp@spawar.navy.mil 

619-537-0145 
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SPACE  AND  MISSILE  SYSTEMS  CENTER 

180  Skynet  Way 

Los  Angeles  Air  Force  Base,  CA  90245 

Mark  Fagan 

Chief,  Modeling  &  Simulation  Division 
Directorate  of  Developmental  Planning 
mark.fagan@losangeles.af.mil 
310-363-2461 

STANFORD  RESEARCH  INSTITUTE  (SRI) 
INTERNATIONAL 

333  Ravenswood  Avenue 

Menlo  Park,  CA  94025 

Ralph  Toms 

Senior  Technical  Advisor 
raiph_toms@sdd.sri.com 

650-859-2852 

STANFORD  RESEARCH  INSTITUTE  (SRI) 
INTERNATIONAL 

1611  North  Kent  Street 

Arlington,  VA  22153 

Greg  Wilcox* 
wilcox@sri.com 

703-247-8467 

TECHNICAL  AUTOMATION  SERVICES 
CORPORATION  (TASC) 

55  Walkers  Brook  Dr 

Reading,  MA  01867 

Murray  Cantor 

Program  Management 
mrcantor@tasc.com 

617-942-2000 

Hal  Jones* 

Technical  Automation  Services 
Corporation  (TASC) 

55  Walkers  Brook  Drive 

Reading,  MA  01867-3297 
hljones@tasc.com 

617-942-2000x2207 

THE  UNIVERSITY  OF  MICHIGAN 

College  of  Engineering 

2236  G.G.  Brown  Building 

Ann  Arbor ,  MI  48109-2125 

Panos  Papalambros 

Professor  and  Chair 

Mechanical  Eng.  &  Applied  Mechanics 
http://www- 

personal.engin.umich.edu/~pyp/ 

734-764-8464 
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TRAINING  AND  DOCTRINE  COMMAND 

Systems  Manager  for  Cannon  Systems 
Commandant  U.S.  Army  Field 

Artillery  School  (USAFAS) 

Ft  Sill,  OK  73503-5600 

COL  Michael  Cuff 

TRADOC  Sys.  Mgr  for  Cannon  Systems 
cuffm@usafas.army.mil 

580-442-6902 

TRIDENT  SYSTEMS  INC. 

10201  Lee  Highway 

Suite  300 

Fairfax,  VA  22030 

Nick  Karangelen* 

President 

nkarang@tridsys.com 

703-691-7765 

US.  ARMY  DIRECTOR  OF  INFORMATION 
SYSTEMS  FOR  COMMAND,  CONTROL, 
COMMUNICATIONS  &  COMPUTERS  (DISC4) 

Pentagon  3E458 

Washington,  DC  20301 

COL  James  Deal 

Executive  Officer 
dealjc@hqda.army.mil 

703-697-5503 

US.  ARMY  LOGISTICS 

MANAGEMENT  COLLEGE 

Attn:  ATSZ  SAM  PQMD 

Bldg.  12500 

2401  Quarters  Rd 

Fort  Lee,  VA  23801-1705 

Dr.  George  Huntley 

HUNTLEYG@LEE-DNS1.ARMY.MIL 

804-765-4265 

US.  ARMY  MATERIAL  SYSTEMS 

ANALYSIS  ACTIVITY 

AMXSY-CA 

392  Hopkins  Road 

Aberdeen  Proving  Grounds,  MD  21005-5071 

Patrick  O’Neill 

Chief,  C4I  Branch 

Will  Brooks 

A2ATD  Program  Office 

Chief,  Simulation  Branch 
Wbrooks@arl.mil 

410-278  -  4946 

U.S.  ARMY  MATERIAL  SYSTEMS 

ANALYSIS  ACTIVITY  (AMSAA) 

Attn:  AMXSY-A 

Rock  Island,  IL  61299-7260 

Jerry  Moeller 

Gmoell@ria-emh2.army.mil 

309-782-7823 
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U.S.  ARMY  MODELING  AND 
SIMULATION  OFFICE 
DAMO-ZS 

1111  Jefferson  Davis  Hwy 
Crystal  Gateway  North 
Arlington,  VA  22202 


Linda  Stone 
Stonell@hqda.army.mil 
703-601-0010,  X  33 

William  Dunn 

Dunnwih@hqda.army.mil 

703-601-0011x25 


U.S.  ARMY  TANK  AND 
AUTOMOTIVE  COMMAND 
USATACOM  TARDEC 
Warren,  MI  48397-5000 

Tim  Bailey 

Attn:  AMSTA-TR-N/002 

Baileyt@tardec.tacom.army.mil 

810-574-5074 

Jerry  Chapin 

Chapinj@cc.tacom.army.mil 

810-574-6144 

David  Holm 

Operations  Research  Analyst 

holmd@cc.tacom.army.mil 

810-574-6537 

Paul  Skalny 

National  Automotive  Center 

skalnyp@cc.tacom.army.mil 

810-574-5436 


Arthur  Adlam 
Attn:  AMSTA-TR-VP 
Adlama@cc.tacom.army.mil 
810-574-8882 
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U.S.  ARMY  TEST  &  EVALUATION 

COMMAND 

Aberdeen  Proving  Ground,  MD  21005 

Dave  Brown 

cdbrown@tecl.apg.anny.mil 

410-278-1473 

John  Haug 

jhaug@tecl.apg.army.mll 

410-278-1275 

LTC  Skip  Paul 
wpaul@tecl.apg.army.mil 

410-279-1480 

U.S,  ATLANTIC  COMMAND 

Joint  Training,  Analysis  and 

Simulation  Center  (JTASC) 

116  Lakeview  Pkwy 

Suffolk,  VA  23435-2699 

Lt.  Col  Robert  Strini 

Chief,  Synthetic  Theater  of  War 
(STOW)  Branch 
strinl@acom.mil 

757-686-7525 

UNDER  SECRETARY  OF  DEFENSE 
ACQUISITION  TECHNOLOGY,  USD(A&T) 
Pentagon  Washington,  DC  20301 

Honorable  Jacques  S.  Gansler 
703-695-2381 

Dave  Oliver 

Principal  Deputy 
oliverdr@acq.osd.mil 

703-697-7017 

UNITED  DEFENSE  LIMITED 

PARTNERSHIP  (CRUSADER  PROGRAM) 
4800  East  River  Road 

Minneapolis,  MN  55421-1498 

Richard  Staiert 

Project  Manager 

Advanced  Systems  Development 
dick_staiert@udlp.com 

612-572-4815 

Bob  Stratton 

Business  Development  Manager 

robert_stratton@udlp.com 

612-572-6257 

Steven  Untz 

Crusader  Information  Mgmt.  Manager 

steve_untz@udlp.com 

612-572-7946 
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UNITED  DEFENSE  LIMITED 

PARTNERSHIP  (CRUSADER  PROGRAM) 
(continued) 

4800  East  River  Road 

Minneapolis,  MN  55421-1498 

David  Wallestad 

Crusader  Program  Director 
dave_wallestad@udlpxom 

612-572-4799 

UNITED  DEFENSE  LIMITED 
PARTNERSHIP 

12443  Research  Pkwy 

Orlando,  FL 

Warren  Richeson 

Director 

407-380-5500 

Bob  Hatton 

Manager,  Orlando  Ops 

Training  Systems  Group 
bob_hatton(g>udlp-orl.com 

407-380-5500 

UNIVERSITY  OF  ILLINOIS  AT 
URBANA-CHAMPAIGN 

Urbana,  IL  61801 

Wayne  J.  Davis 

Professor 

General/Mechanical  &  Industrial  Eng. 
w-davis@uiuc.edu 

217-333-0531 

UNIVERSITY  OF  IOWA 

The  University  of  Iowa  Dept  of 

Mechanical  Engineering 

Iowa  City,  lA  52241 

Dr.  Edward  Haug 
haug@nads-sc.uiowa.edu 

319-335-5726 

VECTOR  RESEARCH,  INC 

P.O.  Box  1506 

Ann  Arbor,  MI  48106 

Peter  Cherry 

Cherryw@vrinet.com 

734-973-9210 
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ENDNOTES 


1.  The  SBA  Industry  Steering  Group  (ISG)  is 
industry's  primary  interface  with  DoD  regarding 
SBA.  The  ISG  is  endorsed  by  the  National 
Defense  Industrial  Association  (NDIA),  the 
International  Council  on  Systems  Engineering 
(INCOSE),  the  National  Training  Systems 
Association,  the  Aerospace  Industries  Associa¬ 
tion  (AIA),  the  Institute  of  Electrical  and  Elec¬ 
tronics  Engineers  (IEEE)  and  the  Electronic 
Industries  Association  (EIA).  Members  below  are 
identified  with  a  *. 


2.  The  Joint  SBA  Task  Force  is  discussed  in  Chapter 
3,  Background.  Members  below  are  identified 
with  a  **. 
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APPENDIX  C 
ABOUT  THE  AUTHORS 


LTC  Michael  V.R.  Johnson,  Sn,  US  Army,  has 
over  20  years  of  operational  and  acquisition 
experience.  His  most  recent  assignment  was 
with  Headquarters,  Army  Special  Operations 
Command  as  Acquisition  Branch  Chief.  Other 
acquisition  assignments  include:  Maintenance 
Engineer  with  Communications  and  Electron¬ 
ics  Command;  Training  with  Industry  at 
Martin  Marietta;  Project  Director  at  Simula¬ 
tion  Training  and  Instrumentation  Command 
(STRICOM);  and  the  first  Project  Director  for 
the  Distributed  Interactive  Simulation  Office. 
LTC  Johnson’s  operational  assignments  in¬ 
clude:  aviation  assignments  in  Fulda,  Germany 
with  the  11*  Armored  Cavalry  Regiment,  at 
Fort  Carson,  Colorado  with  the  4*  Infantry 
Division  as  a  Company  Commander  in  the 
Aviation  Brigade,  Brigade  Plans  Officer  for  a 
Mechanized  Infantry  Brigade;  and  as  the 
Operational  Aviation  Officer  for  the  4*  Divi¬ 
sion,  and  at  Fort  Knox  as  a  Cavalry  Platoon 
Leader  with  the  2/6  Cavalry  Brigade.  LTC 
Johnson  has  a  Masters  Degree  in  Contract 
Acquisition  Management  and  is  a  graduate  of 
the  U.S.  Military  Academy,  the  Harvard 
Graduate  School  of  Business’s  Program  for 
Management  Development,  the  Army’s  Com¬ 
mand  and  General  Staff  College,  and  the 
Defense  Systems  Management  College’s 
Advanced  Program  Manager’s  Course.  He  is 
rated  in  the  OH-58,  AH-1  and  UH-1 
helicopters.  He  is  also  Airborne  and  Ranger 
qualified. 

LtCol  Mac  McKeon,  US  Marine  Corps,  has 
20  years  of  aviation  service,  flying  the  F-4 
Phantom,  A-6  Intruder,  and  the  F/A-18 
Hornet.  His  most  recent  assignment  was  as  the 


Operations  Officer,  Marine  Aircraft  Group- 
31  at  MCAS  Beaufort,  SC.  LtCol  McKeon 
began  his  acquisition  career  as  the  Program 
Manager  for  the  USMC  Land  and  Training 
Area  Requirement  System  at  the  Defense 
Training  and  Performance  Data  Center  in 
Orlando,  FL.  Subsequent  acquisition  assign¬ 
ments  include  Project  Manager  for  USMC/ 
USN  KC-130  Flight  Simulators  and  as  Pro¬ 
gram  Manager  for  USMC  and  DoD  Virtual 
Reality  R&D  programs  at  the  Naval  Air  War¬ 
fare  Center  Training  Systems  Division, 
Orlando,  FL.  LtCol  McKeon  graduated  from 
the  U.S.  Naval  Academy  in  1978  with  a  B.S.  in 
Political  Science.  He  received  his  M.S.  in  Com¬ 
puter  Systems  Management/Information 
Technology  in  1991  from  the  Naval  Postgradu¬ 
ate  School,  Monterey,  CA.  He  is  a  graduate 
of  the  Harvard  Graduate  School  of  Business’s 
Program  for  Management  Development,  the 
USMC  Amphibious  Warfare  School,  and  the 
Marine  Command  and  Staff  Course. 

Lt  Col  Terence  R.  Szanto,  US  Air  Force,  has 
13  years  of  acquisition  experience  at  the  pro¬ 
gram  office,  Air  Staff,  joint,  and  international 
levels.  He  was  most  recently  assigned  to  Head¬ 
quarters  Air  Force,  providing  recommenda¬ 
tions  for  $24  billion  of  the  $450  billion  Air 
Force  Future  Years  Defense  Plan.  In  a  joint 
acquisition  assignment  in  the  Republic  of 
Korea,  he  coordinated  military  technology 
cooperation  programs.  At  the  Electronic 
Systems  Center,  Hanscom  AFB,  he  upgraded 
the  B-1  and  B-52  aircraft  mission  planning 
systems  for  smart  and  conventional  weapons 
capabilities,  and  maintained  238  tactical  air¬ 
craft  mission  planning  systems  deployed  at  150 
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locations  worldwide.  His  first  acquisition 
assignment  was  at  Space  Division,  Los  Ange¬ 
les  AFS,  CA.  Lt  Col  Szanto  spentacaieerbroad- 
ening  assignment  in  F4-E  aircraft  maintenance 
at  George  AFB,  CA.  He  graduated  from  the 
U.S.  Air  Force  Academy  in  1981  with  a  B.S. 
in  Engineering  Mechanics,  and  received  an 
MBA  from  Pepperdine  University.  Lt  Col 


Szanto  is  a  graduate  of  the  Harvard  Gradu¬ 
ate  School  of  Business's  Program  for  Man¬ 
agement  Development,  is  a  distinguished 
graduate  of  Squadron  Officers  School  and  Air 
Command  and  Staff  College,  and  will  attend 
the  Industrial  College  of  the  Armed  Forces 
this  fall. 
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